Since May 22nd, we’ve been working hard on the game in the beta branch, and now we’re back onto the main branch for everyone to enjoy the new items. The full release notes start here and finish here. Let’s see if I can’t summarize that down a bit, though.
The TLDR is that this is almost all prep for multiplayer. If building and testing multiplayer was a car trip, this would be like getting the car in tip-top shape right before you start the trip. If the trip is into rough terrain, then good preparation is going to mean a far smoother trip once you get on the road.
So let’s take a look at that, and then the other bits that are unrelated.
Data Compaction, Many Times Over
Savegames are vastly smaller. They’re also more efficient to write to disk, and to read. This was a massive overhaul, but it also directly effects the way we will be transmitting network data messages.
I’ve written about the benefits of this at length in the past, so I’ll skip rehashing that here. But the short version is that essentially there is a LOT going on in this game, and during networked play we need to be able to fix desyncs on the fly. The only way to really do that is if we can efficiently send large amounts of correction data in the background while not interrupting your main play.
Back in May, I wasn’t feeling really confident that it would work out well. It was borderline, but the processing load and the amount of network packets required was going to be iffy. Any good engineering student knows that you want to build in redundancy: if your bridge can only hold ten tons, and it will have nine tons on it regularly, you’re asking for a terrible accident. You want to start out with a lot more extra capacity to avoid the tragedy, and you also want to do that from the start rather than trying to strengthen the bridge while people are driving on it.
So that’s where we are now: the metaphorical bridge is ridiculously braced and ready for action. This was a really massive undertaking, and it was the chief thing that led to the extended beta period here, so hopefully it makes intuitive sense why I chose to do this before getting into a functional multiplayer alpha rather than mixing the two together.
Game Loading Improvements
This might seem unrelated to multiplayer, but one of the things that we have to be really careful of is that all the players have the same data about game units when they load in.
Because this game was built from the ground up with mod support in mind, there are certain things that I realized could be impairing this consistency. So I put in a bunch of new auto-checking to make sure that things happen the way I expect, and in the process:
- Fixed a few issues that were impacting mainly mod authors, making them have far more xml than they needed to.
- Made the game load vastly faster for everyone (depends on your computer, but for me it loads in 7-24 seconds now instead of 16-90 seconds. The big variance is largely based on how busy my hard drives are and what is already cached in RAM from prior runs.
- Made it possible for mods to adjust base units (like Raider) and then have those modded changes cascade to all of the descendants of that base unit (Daggers, etc).
- Implemented some code that lets us verify that things are consistent regardless of how your OS does the sorting of files in its filesystem. This sort of thing was going to be a potential barrier to cross-platform play between OSes, but it no longer is.
- This also lets you now disable mods and expansions that you have installed from within the game UI, if you don’t want to play with them at the moment. This is key for multiplayer, where you might not have or want the same mods or expansions as someone else you are playing with, and the people with extra stuff can now just disable that in order to play with one another — no need for someone to reach for their wallet just to play multiplayer with you, if you have extra DLC, for instance.
- This prior one also lets us now distribute some mods with the game, but have them be in an “off by default” starting state. This lets various mods be a lot more easily-accessible, without us having to support them in an official capacity or them affecting every game anyone plays. The first mod to be included like this is The Spire Railgun Shop by Lord of Nothing.
Rebuilt Compiler Pipeline
This sounds like a yawn sort of item, so I’ll keep it brief. But it’s something that first Keith and then later I had been working on since 2017 or so, and there were some key limitations with how we were able to link against some of the extra language features that… well, the various multiplayer libraries we’re going to be using (Steam, GOG, and Forge Remastered) all use.
This mean that there was a certain indirection I had to previously build in in order to load those things, and in order to test and change pieces of code in those specific areas it was literally an ELEVEN MINUTE wait for me while I hoped that whatever I changed was okay. Normally it should be a couple of seconds of turnaround, but because of our compiler chain limitations, these specific libraries couldn’t be linked in the direct fashion we prefer.
This was going to make bugfixing in multiplayer an absolute nightmare, as you can imagine, since being able to routinely test things without eleven minute delays peppered all throughout my day is pretty important. If I have a one-line typo, I need to be able to fix that and then see the results within a few seconds, not after 11 minutes. That whole thing was making me dread working on certain aspects of multiplayer, so I put more time into it and finally managed to fix the root problem. Hooray!
Prepping For Efficient Multiplayer Messages
We have long used a very efficient “serialize by index” (instead of name) method to make savegames smaller and more efficient, but that sort of thing could not work in multiplayer because there was no way to have a shared set of agreed-upon indices between the host and client(s).
The game has now been upgraded to handle fully setting those up so that now the host and clients get those synced automatically when the game starts, and so the ultra-efficient smaller messages now work in multiplayer and not just on disk.
Fireteam Specialists
This is a major new feature that Badger created for the upcoming second DLC for the game, which will be called Zenith Onslaught. (Please put away any pitchforks — I have not been working on DLC2 at all yet, as my focus is appropriately on multiplayer. And multiplayer is not something in Badger’s scope to help with).
Anyhow, like fireteams themselves (which were developed initially for DLC1), these new abilities for them have been backported into the base game.
Basically this lets some group, like the AI Hunters, have a sub-group that is chasing just a specific faction. Or just a certain part of your faction. Examples:
1. You’re The Little Fish
Let’s say that perhaps the nanocaust is being absolutely terrifying, and they are a third-party against you and the AI. A faction like them is now able to piss off the AI enough to cause extragalactic war units of the AI to appear. Those are things like the flenser, at the upper levels, where just one of those is absolutely game-ending for you as a player if you don’t have the fallen spire or some other sort of mega faction going on on your own side.
Anyhow, previously we couldn’t use something like the flenser in this fashion, since the AI would eventually probably turn it on you and end the game prematurely and unfairly. But now we can keep the flenser fully focused on whatever aggro’d it so much in the first place — the nanocaust, in my current example.
This means that you can be present in the galaxy while the AI is fighting a much larger foe than you are, using much larger firepower. This is… just plain cool. It also has strategic ramifications, as you’ll want to avoid the planet with the flenser while it’s fighting the other foe, even though the flenser will never fully turn on you and go wreck all your planets. It’s just going to be a fight that doesn’t involve you, but that you have to work around, and the flenser will head back out of the galaxy to deal with its war there once it deals with the target here.
(Of course, the flenser is an extreme example. Mostly you’d see smaller extragalactic war units making an appearance. But mods, and probably DLC2, could aggro the AI enough to cause a really strong faction like that to get into a war that is larger than the one you yourself are engaged in).
2. A Specialist Group Is After Your Treasure
Right now this is used for Major Data Centers (MDCs) in particular. Previously when you captured those, the challenge that they presented was exogalactic waves that would spawn against the planet of the MDC while you were hacking it. If you survived the entire time, then you’d get to keep the MDC.
That mechanic is… overused and kind of boring. For MDCs, we no longer use exogalactic waves at all. Instead, the AI hunter fleet gets a bit of extra budget with some specific objectives to go kill your MDCs. This makes the AI have an appropriate response to you getting the new treasure, but it’s not in the form of this series of kind of hacky waves that come at you over and over. Instead these are groups of specialists in fireteams who are intent on intelligently taking out that treasure when they can. This makes the AI seem much more alive and intelligent, versus just a dump of units at you that you have to withstand temporarily.
3. Changes To Player-Allied Factions
Your allies will no longer go around popping warp gates, which was causing AIP to increase. Instead the AI is now able to use these new mechanics with specialist hunter teams to hunt your allies appropriately. In other words, the AI is now able to mount a proportional response to actions, but without your allies having to do something annoying like destroying warp gates that you might prefer consider existing.
Other Items
- Holy cow, the one bad thing about an extended beta is that it can delay certain fixes. One of those was that all of the AI (and I think Outguard units as well) had the equivalent of mark 1 hull and shields for the last couple of months if you weren’t on the beta branch. That’s going to be a bit of a jump in difficulty on campaigns that were vanilla if you were on the last non-beta one all this time — apologies.
- There were a bunch of other bugs fixed that were less likely to affect everyone, but which were still annoying and which are great to finally have on the main branch.
- A bunch of achievements that were not triggering correctly now do.
- The Parasitic Starting Fleet has had its balance improved, and should be more attractive to start with now.
- AI ships from spire debris are now a lot more dangerous.
- The Imperial Spire Fleet now gives you vision on their planets so that you can watch the fireworks.
- Multiple copies of the same faction type (whether they are friends or enemies to one another) now try to spread themselves out in the galaxy to make things a lot more interesting. If your playing with good and evil marauders both at once, then they won’t be neighbors most of the time, now.
- We improved how we are doing our fixed-integer math, which fixes some math rounding issues that could lead to a bit of odd data with certain bonuses on the UI in particular.
- Fixed a very unusual bug where the AI was basically able to get frigates “for free” as part of their reinforcements in the rare cases that they actually used them.
- Make the Marauder less likely to suicide themselves against Praetorian Dragons.
- During gameplay, when you click into the Galaxy-Wide Options screen it now shows you ALL of the values, not just the ones you can edit.
As with the factions screen, it just has the ones that are non-editable as text values, but you can examine your settings this way and also see their tooltips. - If the game runs into an exception while trying to start up, it now gives you a visual error rather than the loading process just seeming to hang.
- The Nanocaust and Dark Spire shouldn’t be able to rebuild a Nanobot Center or a Locus for at least a few minutes after a previous one on that planet was destroyed.
- Cross planet waves against a specific faction now will stay focused on that faction (and not go off and attack the player).
- The “brighter color” hex and color for faction teams now uses new logic by -NR-SirLimbo to make sure that faction names should always be nicely readable even if their actual colors are normally very dark.
- Various more voice lines have been integrated into the game.
- The game now has a Notification for when Dark Spire Loci are warping in.
- It turns out there were some sort of edge cases where the AI king could be dead but its faction would not have gone through the “we have lost” logic. We don’t yet know how that could possibly be possible, but we now have a safety check from then on in the future that makes sure that even if the kings death was somehow missed when it happens, it now recognizes that it happened in the next few seconds and will mark the faction as defeated.
- Fixed a SUPER longstanding bug where it would sometimes write “Was looking for a wormhole (some stuff) but couldn’t find one.” This was also causing some bad ship behavior. It’s been around, but rare, for years now.
- Fixed a “fun” bug where some enemy ships with fireteamss would be unable to decide between targets; they would get partway to one target, then turn around as if going to the other target. This looked like a Buridan’s ass style problem with the targeting code. The actual problem was that Fireteams were incorrectly declaring themselves winners of a battle without showing up.
- Fixed what appears to have been a relatively old bug (somehow?) where the max strength of a lot of fleets was not being properly calculated. It seems like this may have slipped by for a few months, or since a bit before DLC1 somehow.
- Also fixed a bug that probably goes back to the start of fleets, where instead of the current strength of fleets on the fleets sidebar, it was showing their total strength. For command station fleets, it was showing a max that they could not even hit, too.
More to come soon!
But here’s some further reading on what we’re doing in detail:
Multiplayer Schedule, And Why
It feels like an endless wait for multiplayer, doesn’t it? I hear you on that — I think it probably feels even longer for me, since I’m the one working directly on it daily and it’s been so many things.
I wanted to take a minute to step back and address WHY things are taking so long. You may or may not know this, but we actually had fully functioning multiplayer for this game back in 2017, before turning it off because we couldn’t reliably try to keep sync in the game while building out so many features like we were doing until late 2018.
It seems like it should have been a simple matter of just turning back on that old code, right? Well… unfortunately as we developed the game, it got a lot more complex.
Essentially, the first AI War existed in a perfectly deterministic state where, after initial game sync, nothing had to be sent between clients and the host except for commands from the players and the higher-level AIs. It also sent some basic check messages to make sure that results were not diverging on your machines — and if it detected a divergence, then it would flag that as a desync and stop you from playing until you saved, reloaded, and had the clients reconnect.
That determinism was achieved through a lot of really rigorous coding practices, and using fixed-integer math instead of floating-point math, and only using one thread for the main simulation.
In AI War 2, we can’t really enforce how rigorous modders’ code is; we also are using many threads, all of which are designed to be deterministic but which… turns out sometimes are not because of exceptions that we can’t solve in a deterministic way without slowing the entire game way down; and we have to use some floating point math here and there (forcefield pushback, for one) in order to get the accuracy and performance the game needs.
So what this means is that the entire premise for multiplayer in AI War 2 got more complicated. The game is MOSTLY synchronous, MOSTLY deterministic, and does a great job at running blazing fast. But desyncs can and will happen, probably with some frequency depending on how different your PC specs are from the people you are playing with (or what errors in rigor have been made in mods you want to use), and the old approach of stopping the game and making you reload and reconnect is absolutely unacceptable. You could be running into those sort of issues once an hour, or once a minute, depending on the circumstances.
That means that, while we strive for determinism as much as possible, we also need to have automatic desync repair. This is familiar more for action games (which exist in a state of constant desync by nature) rather than strategy games. I have experience with this from the A Valley Without Wind games, but they have an incredibly small fraction of the amount of data that AI War 2 has going on. It’s a much less intense environment as a multiplayer coder.
Anyhow, this has meant that the environment has to be extra well-prepared for sending efficient messages, since there will be more of them. It also has to be able to fix small problems in a spot-fix fashion without re-syncing the entire game.
These added challenges meant that I needed to restructure some substantial bits of the code that were working perfectly fine if you look at them from any other context. I still have some areas to deal with for that, but the list is getting much smaller.
We’re still on track to have multiplayer “in the next few months,” but I realize I’ve been saying that for months now. The really good news is that alpha of multiplayer should be way more smooth, and a lot shorter of a period before we can get to the point of it being fully polished.
I’m keenly aware that as soon as folks can play multiplayer, they will do so; and if it’s a broken mess on the first alpha, and it takes me months to make meaningful improvements to it, I will lose my testers. So I’m being extra cautious and hitting all the areas that I expect I will run into before even starting the first alpha. That should ultimately be a lot less stressful on me in a time-pressure sense once we start doing multiplayer alpha.
What’s Remaining For Multiplayer?
So what is the todo list, then? It’s getting shorter!
- Now that the compiler chain is improved, I need to get Steam and GOG linked in a better way, and also get Forge Networking linked in that fashion. This will be FAR easier than what we were doing before, and more in line with how those respective libraries expect you to use them. I’m very excited about this being possible, finally.
- The actual cycle of multiplayer messages being sent and received, and that keeping the game flow going, is done and has been for years (literally since early 2017), but I need to reintegrate that into our core dlls and improve the efficiency of some bits of it. It’s not large amounts of code, thankfully.
- I need to redo how the initial world state is sent to new clients joining in, but that will also be quick — thanks to the rework of world save data styles and sizes.
- Our old style of desync-detection from 2017 has been stripped out now, and I need to add in a new kind of rolling desync detection that will find problem entities and then allow us to fix them. To keep this fast and relevant, I’m only going to care about a few things: position, health, and shields. If those match, then probably everything else is close enough. How to handle the time differential on this is really challenging, and only in the last couple of months did once facet of this occur to me. I have a couple of solutions in mind, but it’s going to be… “fun.”
- The way that we send information in multiplayer is by GameCommand, and right now those are generic and bloated (they are overly multi-purpose). I am going to replace those with a new set of custom GameCommands that us developers and modders can use as-needed and have them take full advantage of the new smaller data sizes. Fun fact, this will also make single-player run very slightly faster when a lot of things are going on.
- The really big one that remains is making sure that the cross-machine identifiers (PrimaryKeyIDs) are consistent between machines. I don’t fully have this figured out yet, but I think that the interim state of it will essentially be that there are occasionally too many messages being passed around because of rolling sync errors. I will probably punt this issue into something I look at during the alpha, so that I can gauge what sort of impact it really has on performance, and where the problems are coming from.
- After that it’s more or less a matter of making sure that the lobby fully works as we expect, and various other small UI systems to get multiplayer basically playable. A lot of work went into the lobby in particular in the last few months to make that as close to as ready to go as possible.
- There are then whatever changes we need to make to balance in order to make things “feel right,” which will be a matter of working with the multiplayer alpha and beta testers. A lot of things we already did in the past, like making science collection a humanity-wide thing that each player gets a copy of, rather than something people have to do individually (what a pain that was in AIWC). We will have to scale waves like we did in AIWC multiplayer, or in some other fashion. But a lot of the difficulty scaling is inherently handled by AIP being higher when you have to take more planets in multiplayer.
So, to sum up the plan for the short term:
- I need to re-link the networking libraries using the new compiler chain. This might take a couple of days. I’ll do it on a beta branch in the next week or so, at least for a bit, because it will potentially break the ability to log achievements on some machines or OSes until I get it fixed back properly (assuming it doesn’t work on try one).
- Then I will pull the network-loop code back into the main game, which is a part of one day job.
- Then I need to generalize that to work for all three networking code paths (Steam, GOG, Forged), which probably will only take a couple of days at most.
- Then I need to re-code GameCommands to be more efficient and special-purpose. This is probably a job that is a couple of days long, and will potentially lead to widespread bugs for a week or so after it.
- Then I need to implement V1 of the desync detection and correction code, which should probably only take a few days.
- Then I need to get the basic connection interfaces working, so that you can choose to connect to players via Steam, GOG, or a direct IP address (forged). This will be a moderate challenge, because Steam and GOG both have APIs that I’m pretty unfamiliar with in this exact area. Once the connection is going, it’s much easier. But with the new compiler chain, at least I can hopefully get these going in a matter of days or a week rather than it being something that takes multiple weeks.
- At that point we should be able to reasonably declare that the multiplayer alpha is ready, as you should be able to connect to friends and play with them, and the desyncs would cause network performance degradation rather than something permanent that your game can’t recover from.
- Then it’s a matter of the other features on multiplayer, mainly regarding things like donating fleets between one another, and/or whatever else we come up with that is desirable.
- Oh, and during that period if we’re seeing network degradation or other issues due to the constant need to sync errors, then that will be to be investigated and improved. But those things are most of what the focus of the alpha/beta will be on.
How long will each of those line items take in reality? I guess it depends on what gotchas I run into that are unexpected at the moment, but you can always watch the release notes to see how things are coming. A lot of the really hefty things that required major reworks of the game are done, so that’s a win.
There are some things that I’d probably prefer to do sooner than later, like allow you to enable and disable expansions/mods from the main menu without a restart of the game, so that might take a chunk out as well. But it should make things easier on everyone as they get more into multiplayer.
Please Do Report Any Issues!
If you run into any bugs, we’d definitely like to hear about those.
The release of this game has been going well so far, and I think that the reviews that folks have been leaving for the game have been a big help for anyone passing by who’s on the fence. For a good while we were sitting at Overwhelmingly Positive on the Recent Reviews breakdown, but there have been a lot fewer reviews lately and so that has definitely had a material negative effect. Go figure. Having a running selection of recent reviews definitely is helpful, but at least we have a pretty healthy set of long-term reviews. If you’ve been playing the game and enjoying it, we’d greatly appreciate it if you’d drop by and leave your own thoughts, too.
More to come soon. Enjoy!
Problem With The Latest Build?
If you right-click the game in Steam and choose properties, then go to the Betas tab of the window that pops up, you’ll see a variety of options. You can always choose most_recent_stable from that build to get what is essentially one-build-back. Or two builds back if the last build had a known problem, etc. Essentially it’s a way to keep yourself off the very bleeding edge of updates, if you so desire.
The Usual Reminders
Quick reminder of our new Steam Developer Page. If you follow us there, you’ll be notified about any game releases we do.
Also: Would you mind leaving a Steam review for some/any of our games? It doesn’t have to super detailed, but if you like a game we made and want more people to find it, that’s how you make it happen. Reviews make a material difference, and like most indies, we could really use the support.
Enjoy!
Chris
