Month: February 2017

Rethinking the icons…

I already complained about the icons today, and there are a few things going on with them that are definitely moving in the right direction.  And I’ve figured out the math on how to handle them even better.

However, Blue and Cinth and I were talking today, and there may be a better, more attractive way for us to handle the icons in general.  We still need to do some mockups, but basically it would involve making the icons 2D and trying to keep them feeling like HUD elements that move around, tracking the squads, rather than elements in the 3D world.

For reference, the center of this screen is more or less the general style of GUI we’re going for:

Bear in mind, of course, that’s still incredibly early as a mock for the GUI.

In this new concept for squad icons, people will see those as basically being “part of the GUI,” kind of like how Iron Man would see targeting icons laid over top of real-world objects through his helmet, if that makes sense.  We’ll see if it works out visually or not, but in theory it might.

Load-wise, we can probably do it in a way that is approximately equal in cost to rendering the 3D icons the way we’re doing them right now.  In theory.

Luckily we have a ton of testers now, so we can easily do A:B testing on a wide array of hardware and OSes, right?  Such an awesome resource to have you folks. :)

Anyway, we’re going to do some prototypes of the bomber and fighter, and then see where that gets us.  Worst case, we keep with the existing style we’ve already established, minus being so crazy over-glowy.

I already have plans on changing up how squad selection visually looks, so this might fit well with that.  The actual visuals for the ships themselves are already nicely set and we’re cranking through them, but there are various other areas of the game that aren’t nearly so far along.

  • Obviously the GUI itself is just completely test junk right now in the actual game.
  • I had thought the icons were pretty final, but we think we might be able to do better.
  • The planets are pretty final, but I’m less happy with the background spaceboxes at this point and want to redo those if possible.
  • I’m not really happy with the shot graphics at all, but that’s not really a crisis because it’s just the first one anyhow.  That’s right up my alley in terms of making a lot of that stuff look amazing.
  • The wormholes I created are something I hate, and both Cinth and I have been poking at ways to do a replacement.
  • The ship contrails are reasonably final unless we think of something better as a way to handle that, although I’d had concerns about performance.
  • The particle effects for ships exploding and similar are nice, but I don’t see those as being final anymore.  I had thought they were, except needing to be scaled better for the types of ships in question… but now I want something more dramatic, and have a good idea of how to do it.

Overall I’ve been getting a much better idea even since just yesterday of how much room I have in which to play with visuals before it starts making some hardware unable to be used.  I’m really pleased with early results, since so far I have a bit more room to play than I thought.

Anyhow, apologies for not having the next release ready today.  I will definitely have that out tomorrow.  Thanks also for the torrent of early feedback, it’s been really useful!

Thanks, Amazon…

We’ve been using for years (since 2011 I think), and I can count the number of times they’ve had downtime on one hand…ish.

But today, the day after our alpha starts… naturally we get this:

Keith notes: “I’m particularly fond of We have now repaired the ability to update the service health dashboard.” From the AWS side.

Not sure how that’s going to impact our ability to get a new build out today; we’re still working away at things, but we can’t check in code or share it between us as of about 3 hours ago, and this outage is lasting longer than usual.

Working on icon colors and flicker.

Cinth has been complaining about the neon signs that are the main icon colors for some time now, and many others have also brought it up.  Just haven’t gotten around to it yet, until today, since he kind of prompted me to do some more work with it.

Ultimately he wound up making a goodly selection of new things down in the bottom rows there (the topmost row is mine, the rest are his).

These are easily moddable, by the way, so technically Cinth made the “first mod,” although that’s dubious since it’s just part of the main game now since he’s directly helping create the game… but he wanted to note that, heh.

Anyway, the topmost row is actually after a whole lot of work on my end, working on completely new shaders today, too.  So that row is a lot better than things were before.

That Stupid Flicker!

Problem is, we’re still getting an absolutely awful flicker at times with the icons in general.  The problem overall is z-fighting because of floating point precision issues on your GPU at far distances despite us scaling these things up.  The glow exacerbates that.

What I’m going to wind up doing is recoding these things so that they don’t exist in the same space, but instead so that they exist in a layered camera that uses fake distance, and uses no rotation whatsoever.  By faking the distance, I can minimize chances for floating point errors, and by avoiding the rotation I avoid any form of rotation-based strangeness.

This will mean that I need to translate from the target point of the icon in world space into screen space, then back over to “other camera world space, with fake distance applied,” but that’s a pretty cheap calculation and I don’t have to do it every frame.  In the end, it’s probably cheaper than some of the math that I have to do now in order to make them billboard, and it will solve the flicker.

Frustratingly, I have bigger fish to fry right now.  So I’m not sure if I’ll get to that this week.  But I wanted to let folks know I figured it out, at least.

Okay… what!? OSX not only works, the performance is ridiculous so far.

Last time I fired up AI War 2 on my Mac, which is a pretty outdated machine with a really really bad GPU, the results were pretty bad.  I wasn’t really surprised, because I was running it with all the settings turned up, and the music was throwing errors a bunch, and I knew it could get a lot better with some polish.  An old intel 4000 integrated GPU is not exactly at the middle end of the market anymore, even.

Just retested the latest build, which has the music swapped over to a different playback system, and which has a metric ton of performance optimizations from myself and mainly from Keith (the entire game simulation runs on a secondary thread, now).

Results? 300-some odd ships in the very opening battle, with all the settings turned up to maximum still, and it was buttery-smooth.  I have no idea what the framerate was, but I feel confident saying it was north of 60.

I just… whaaat?  That’s… not what I expected.  It’s fantastic, but it’s also a big surprise.

I’m not going to run out immediately and lower the minimum stated system requirements expectations, but that is certainly super duper positive as an indication thus far.

Reminds Me Of Something…

As kind of a fun aside, back when I was coding AI War Classic in the early days I was hoping to get 10k ships in a game, and was pretty sure I could do it.  The maximum any other RTS had at the time was about 1000, and it was super choppy and laggy with that many units in the other titles.  I felt like I could bump that up by a 10x partly thanks to going 2D, and partly just due to coding practices.

By the time the first public builds were around for AI War Classic, the typically number of units in the game was more like 30k, and it would start to chug around 45k units.  Within a year after that, 50k was more typical and it would start to chug at 75k units.

More Generalized Thoughts on Optimization

Sometimes it’s just really surprising how things can go when you throw everything and the kitchen sink at optimization.  It’s kinda-sorta working, it’s getting along okay, and then suddenly you pass this critical mass and whoosh the performance jumps by an order of magnitude.

I’m not remotely ready to say that’s happened here, yet, though.  The simulation is not remotely finished for the game, as there’s still a lot more AI to build out, and lots more ship functions.  The largest battles still have only involved just a few thousand ships at a time for me, whereas in classic sometimes north of 10k ships in one fight would happen.

So I want to see what happens with all those things.  Right now early indications are ridiculously, surprisingly good; but some monkey wrench could still very well appear between now and early access that makes me say “yeah, the minimum system requirements for a truly pleasant experience should still be more than a lousy intel 4000.”  Then again — maybe not. :)

Oh, One Last Thing

In AI War Classic, the simulation speed was locked to the framerate on the slowest computer in a multiplayer game, or to your framerate in solo play.  Kinda common for RTS titles, though not all of them.

In AI War 2, your visible framerate is completely unrelated to simulation speed.  We’re actually running the simulation speed at 10 cycles per second no matter what, which gives us a lot of muscle per cycle on a secondary group of threads.  It doesn’t need to be any more fine-grained than that.

Then for input and actual visual display, of course it can run at much higher framerates, basically up to whatever your hardware can handle.  If you’re running at a buttery-smooth 200fps and everything is just peachy, and I’m having more trouble on my machine at a hard-won 30fps or something, the simulation won’t slow down for either of us.

I’m a pretty happy guy about now. :)

A fantastic conversation.

Just general forum humor.

BadgerBadger: What does “lerping shots out of ships” mean? I know larping, not lerping.

Keith: Please, please don’t let the shots larp.

Cyborg: The AI will now say, “Lightning bolt!” before each attack.

Chris: “I’m attacking the darkness!”

Keith: 20,000 MkV Gazebo Guardians to Murdoch in 0:32…

Chris:  My favorite version of that skit:

Hey, do you remember the Gazebo monster we put into Valley 1 as a bit of an easter egg? Most of the Gazebos were harmless, but a tiny minority were absolutely deadly murder machines.

I completely forgot about that. :)

No real way to build asset bundles selectively in one unity project??

That’s pretty frustrating.  It’s not a dealbreaker, because for truly isolated things (music and sound effects, for instance), I can just make yet more unity projects.

But this means that in order to have some platform-agnostic asset bundles for music and sound separately from everything else, I now have to have FOUR unity projects for AI War 2 rather than just the two we had before.

Seems kinda like an oversight of something obvious on the Unity 3D advanced build pipeline process, but at least there’s a trivial workaround.  Spent more time on that today than I meant to, but it was legitimately puzzling.

Writing now while I wait for things to compile to their bundles for the first time, ever so slowly.

Definitely having to prioritize my time.

If you’ve looked at my todo list, you’ll see why.  There are so many things that I want do do, where I just think “oh, but I could get that done in just a couple of hours at most!”

And it’s true in each instance, but there’s a very real additive cost to all that.

I know that I can make the background starfields, which I’m growing tired of already from a visual sense, look a lot better and more stylized.  But is that more important than bugfixes?  No… sigh.

I want to get MasterAudio set up so that I can go ahead and have music ducking and so forth in place when I get to that point… but is that more important than just getting uAudio out and thus the game working on OSX properly, and then back to those bugfixes?  Again… no… sigh.

And so it goes.  Things move fast with Arcen, but there’s still just never enough hours in a day.  I’ve had a very unhealthy work/life balance for years, and in the last year or so I’ve been trying to get that more under control.

So… priorities.  Making sure I can load music in a cross-platform way and play it efficiently.  We’ll worry about audio ducking later when we’d even be doing that at all.

Among other things that will just have to wait.  At least with the todo list, you alpha folks can see where my head is at and what I am valuing over what in the short and middle term.

Hopefully everything on that entire list is done within the next month, knock on wood.  I have three in which to do it, but I’d rather be done in one.

To-Do List

Figured you might like to be able to see my todo list.  I just moved it out of my notepad file and into trello, so everything is listed as just being added.

Some fun performance notes from Keith today

I don’t think he’ll mind if I share these tidbits from his victories.  All of the below is written by him:

Commit Notes

– moved main sim execution to separate thread- moved most of the “ui reading the current gamestate” stuff to that thread too

– moved the generation of sync data to that thread as well

– moved the update-visual-object stuff outside of process-sim-step; still on the main thread of course but now it’s easier for you to profile it without sim stuff cluttering up the results

– various performance improvements, some of which are fairly significant


So right now there’s some volatility in the visualization code but it’s not causing problems right now since I think it’s mostly dealing in primitive types. Will need to be dealt with later but doesn’t appear pressing.

Performance-wise the sim’s burden on the main thread is almost gone during most frames. There’s still some spikes when a lot of entities die at once, as that’s a lot of list removal work. I can actually optimize that a bit more, but for now with the sim-on-the-main-thread almost always under 2ms (and often way under that) I figured this was ok for now :)


Oh, I should also mention that the main thread per-frame heap alloc is also almost gone, and most of that was eliminated rather than just shifted elsewhere.

One Hour Later

I’m currently working on the other-thread heap alloc, via the deep-profiler and setting SimPlanningLoop.IsOnMainThreadForProfilingPurposes to true. It just makes everything execute sequentially, rather than farming it out.

Another Hour Later

Just nuked the heap alloc of the short term planning threads (which do most of the work) from about 800k to about 40k; I can cut about 25k more, too, just needed to go now. The remaining looks like legitimate allocation on the partition list (used for find-everything-within-range checks), and that should in theory stop once the list is big enough that all future checks use less space.

The execution thread is still doing 300k+ per go, and I’ll take a look at that later. Not sure what the long term (AI) thread is doing heap-wise, haven’t seen it in the results yet.

uAudio dies, and we move to more asset bundles.

One of the things that I recently discovered was that asset bundles for meshes seem cross-compatible for at least Windows and OSX, but shaders for sure – and possibly materials – are not.  The shaders bit I knew; DirectX9 shaders aren’t going to work in OpenGL.

So that’s wound up with us having to have separate asset bundles for each of the three platforms, which is a minor inconvenience but no big deal beyond that.

What is a bigger deal is that we’ve been using uAudio in order to stream mp3s off disk, and that doesn’t work on OSX.  It was supposed to, I’m pretty positive, but anyhow it’s busted.

For that, I’m just going to throw my hands up and move elsewhere.  I wanted a more unified audio pipeline anyhow, so that I can do some proper music ducking with sound effects playback integration once that’s in.  uAudio was already going to complicate that for me (note that that’s one of a few reasons there are no sound effects, despite their being music, in the initial alpha of the game).

At any rate, I’m going to create a separate asset bundle for audio in general, and my hope is that this one can be cross-platform, which would be really nice.  It won’t make much difference to you, but it saves me time uploading to Steam, and it gives more common ground between the different builds.  Right now there’s almost a gig of stuff that is OS-specific, which is annoying.

Some folks were complaining to me about not wanting to support mp3 on principle, so I guess that wins out. ;)  With things in asset bundles, they’ll be in a streamable intermediate format that I believe is ogg.  This will mean that in order to get the soundtrack you have to actually get the soundtrack, though; you won’t be able to just find it in your folders in the game install anymore.  That’s an unintended side effect, but it will improve performance during gameplay, so there is that…

My old approach of reading oggs directly off disk was just too low-performance and memory-eating.  The asset bundles approach will help preserve speed, which is a big deal in my opinion.

Hitching? AI War 2? Keith says: “Ha.”

Well… okay then!  Keith already moved the simulation off the main thread in the course of just this morning.  Unbelievable.  On my main machine, a GT72VR GRE Dominator Pro (so GTX 1070 and latest i7), I’m sitting pegged between 180fps and 200fps during the opening battles with all the settings cranked up and 8x MSAA and so on.  Almost no giant lag spikes – amazing!

What is kind of odd is that now there are some GUI.Repaint spikes every so often that take 35ms to execute, but they are only every few seconds at most from what it seems like.  I’m not sure what the deal is there, but it’s something to do with Unity’s GUI internals.  I’m not happy about it, but one long frame every few seconds is not something I was really aware of until I did the profiling and saw it.  I wasn’t noticing it while playing.

Strangely, the occlusion culling spikes seem to have disappeared, too.  I don’t really trust that, because if they disappear for random reasons then they can come back for random reasons, too.  But it makes it no longer an imperative thing that I try to fix that in the next week… since right now there’s nothing available to even fix.  But when it comes back… I’ll be ready and waiting. ;)

Anyhow… way to go, Keith!  I’ll have to retest this on my mac later today and see how things feel now.

RIP My Linux Install

Okay, so… “funny” story?

I’ve been mostly running ubuntu in a Parallels VM on my Macbook.  It’s worked well for the last number of years.  However, now… well, it’s a combination of drivers and just other accumulated issues.

Basically, the short of it is that my linux install still works, kinda-sorta, but it’s buggy as all get out and randomly crashes and is constantly reporting errors in itself, steam, and various random programs.

This VM was perfectly capable of running Release Raptor, at least at a low framerate with a lot of quality settings off.  It also ran a much earlier build of AI War 2.

However, now it literally gives a blue screen in the window that unity pops up when I try to start AI War 2.  I think that this is because my VM is so hosed in general, and also because it’s running an older version of ubuntu than unity now supports… and which I can’t upgrade from due to lack of virtual disk space.

All of which is to say, while we DO have “day one alpha” linux support in the game, as promised, I am not presently sure if it works at all.  I’m tired of VMs, and I work pretty much exclusively on laptops these days, so what I’ve decided to do is order a new Oryx Pro from system76, with ubuntu natively installed.

All my other machines are MSI, Asus, or Alienware at present, aside from my mac and a trio of old desktop machines in my attic.  I don’t have room to run a bunch of desktop towers in my office, though, which is why I went the VM route in the first place.

I looked into getting ubuntu on my existing laptops, but it’s just one of those things that is so fraught because of lack of driver support from those vendors in a lot of cases (shakes fist).  Trying to test the game on those would involve hours of actually setting up the ubuntu install, possibly corrupting my master boot records (thanks, Windows 10), and possibly still having a steaming mess of unsupported hardware.

I figured I should Do This Right, instead.  The bad news?  It may be another two weeks before that new linux machine actually reaches me.  So if linux works out of the gate for you during alpha, then I’ll be overjoyed and that’s a relief.  If there are problems, I can try to use your error messages to figure out what is up, but most likely I’ll need to have my own linux box before I can really get into it if it’s something complicated.

Hopefully it just works.  But you know how that sort of thing can go.

I apologize for not being more on top of the linux build from the start, but it WAS working and it hadn’t occurred to me to test it lately and find that it was suddenly not.  The way it dies instantly pretty much guarantees that it’s missing something in my particular install, and not our code doing something that breaks linux.  My worry is more that if you get past the first issue on your machine (having a non-broken install and all that), that you’ll instead run into something that’s broken in my part of the code.

Anyway, teething pains.  Apologies again if it doesn’t work day 1, and fingers crossed that it will.

AI War 2 on my Mac

My Mac at present is an aging MacBook Pro, Retina, 15 inch, Early 2013.  I’m still running 10.10.5, because I like to stay kind of out of date on that for testing purposes.

Worth pointing out for anyone not up on that hardware: the GPU is a measly Intel HD Graphics 4000 1024MB, and it’s running at 2880×1800.  Tall order!

The GTX 1070 on my main windows machine is over 2600% faster than that little intel.  Whew.  Honestly that machine struggles REALLY hard to run Minecraft.

I also have a spare machine, this one running windows 8.1, that has a GTX 650M in it.  I’m going to be benchmarking on that one, too – and it’s only about 145% better than the intel 4000.  Still substantial, and it never had any trouble with Minecraft.

Anyway, overall from a first test on my mac with the various recent vis-layer performance improvements (but before those of the simulation thread today), I was pretty pleased with the results on my Mac.  I had all the settings cranked all the way up, and while it was choppy it wasn’t a complete slideshow.  That was at a high resolution, as much MSAA as it could muster, all the post-processing effects, all the special effects on max blast, full quality textures, etc.

The result wasn’t something I’d really want to play in terms of how it performed, but looking at that from a worst-case scenario I’m happy with how that was starting out.  Cinth isn’t even done setting up quite all the ship LODs yet, so some of the vis layer was needlessly harsh, too.

I think some of the hitching was also due to music playback trying and failing to play, repeatedly.  That may have been the biggest slowdown, I’m not sure.

Possibly if I get this updated to a newer version of OSX and see if it can run Metal, then I might be able to use the BC7 file format for textures, which right now is being transcoded to something else (possibly native uncompressed RGBA, which would really slow things down in the GPU bus).

For anyone already on a newer version of OSX, probably Metal support will “just work,” so long as your hardware supports it.  I need to do a deeper dive into some of that, but I feel quite confident I can get this running in a way that is acceptably playable on my aging MacBook.  It’s going to be one of those things that continually evolves over the next few months leading up to Early Access.

Hitching and Jitters In Early Alpha AI War 2

We’re headed for alpha of AI War 2 on Monday, and that’s for the people who want to get into the game “earlier than early,” which I’ll write more about later.

One of the things I wanted to talk about right now, though, is some hitching and jitter that will likely still be present to some degree in that first version.  This is an issue that is cropping up this week due to two factors:

1. The simulation is becoming more complex, and thus more costly to run.  It runs every 100ms, so that’s a hitch with an extra-long frame there.

2. The visualization code runs every frame, so usually 60-140 times per second on my machine, and so there’s a huge difference in the timing between the two sides.  It’s only recently that this has become efficient enough for me to notice such a huge difference between the two groups of frames, though.

In other words, things were more balanced-out previously because we hadn’t finished certain optimizations AND because the simulation was still growing.  Now the simulation has grown more and other optimizations are in, and so the jitter rears its head.

I’m writing this so that hopefully nobody freaks out on Monday. ;)  It won’t take us long to resolve this, but that’s one of the casualties of being in “earlier than early.”  This is part of how the hotdog is made that you get to see.

Last post, I wrote about some improvements made to the fixed-int math as well as RNG calculations for noncritical bits.  Those help the jitters noticeably.

Overall what we’re finding, though, is that we simply need to move the main simulation off the main thread of the game (it’s already using a lot of worker threads, but now the core loop also needs to go).

That was already on Keith’s todo list, but it got bumped up in priority in the last day or so.  I don’t think either of us have any idea if that will be ready by Monday or not, but for the sake of safety I’m going to just assume not.

Occlusion Culling!?

The other thing that is causing periodic hitching is Unity’s own occlusion culling logic.  It occasionally jumps in with a 28ms frame, once or twice every two seconds.  That’s incredibly frustrating.

That said, I found something kinda-sorta similar when I was coding Release Raptor, and I implemented my own frustrum culling solution that I had intended to port over to AI War 2, anyway.  So that’s coming up soon, though likely not prior to Monday, either.

Occlusion culling, in this case, is basically just saying “hey, the camera isn’t pointed at me, so don’t send me to the GPU to render.”  Unity has to calculate that sort of thing for every individual mesh that is rendered if I use their version.  In my own version I can calculate that for things like entire squads.  That cuts down the number of calculations enormously, and I can also calculate AABBs for the squads vastly faster than unity can for individual meshes.

That right there is a good example of the differences between generalized solutions (which must work in any game without knowledge of context) versus specialized solutions (which really wouldn’t work in general cases, but are incredibly efficient in certain specialized ones).  It’s always about choosing a mixture of the two approaches, and right now the generalized approach is biting us in the rear more than I expected.

Particle System Updates!?

This one I don’t even know what it is, and I haven’t had time to investigate it yet.  But every so often – I’m not even sure with what frequency – Unity is throwing a slow frame (again just under 30ms) where is is doing a “particle system update.”

Yes we’re using their shuriken particle system, but why it suddenly has to go through some sort of cleanup or… whatever it’s doing… I have no idea.  So that’s another thing for me to investigate in the coming weeks.

GC? Nuh-uh!

In AI War Classic, the garbage collector running periodically was the biggest source of hitching.  Thanks to our massive refactoring, we’re generating very little waste in AI War 2.  There are some errant GC allocs that we’ll clean up later (you’d like an actual game first, we presume, heh), but those are all very minor and would fall under the banner of premature optimization right now.

At the moment, we’re keeping things clean simply by using clean coding practices and our hard-won knowledge of strange things that cause mono (but not .NET itself) to generate garbage (see: foreach operator).


Figured this was worth a writeup since some folks are going to be getting into the game soon, and the first question I would have is “what’s with this hitching?”  Ironically, the worse your computer is, the less hitching you’ll have right now. ;)

But anyway, I figured an explanation of what it is as well as why it’s going to be going away very-soon was warranted.  Cheers!