Magic Item Tech, testing – part 3.

Hi all!

It is time for some software testing framework talk again 🙂 !

As you know, I’m a big quality/testing advocate (especially when it comes to software!!!), and I’ve been working for a while now, in my “lab”, on a high-level testing and automation framework, since I’ve reached the limits of usefulness of unit testing my game projects.

I’m going to showcase the newest addition to this testing framework, so if you completely missed it and interested, here are the two older posts summarizing the design and some implementation details ( with showcase video 😉 ) about it:
Testing – part 1.
Testing – part 2.
A quick recap, if you would not like to read through the usual pile of text 🙂 , but still interested in this dev. log. entry:
It is a “capture and replay” based framework, where you can record (or edit/create) input events (or special ones, e.g.: network, debug events etc…) from the game to replay them later. For checking certain functionality, the replay files can be filled with assertions targeting the properties of any game-object within the game world at any given frame of the replay.

So the system served me well while developing the Operation KREEP update. It took less than two work days to reach a pretty high coverage of game-play features and the UI work-flow (around 80% code coverage in 12 hours). This test-suite helped me a lot while releasing the Steam version, besides keeping my sanity by covering and assuring, that most of the high level features work, it saved me from introducing a few pesky bugs while coding the new features!

After a while though, execution times grow as it is to be expected. The 67 replays for the final Steam build(s) which checked the UI work-flow, completed the tutorial, played till getting most of the achievements and so on and so on, requires around 11 minutes to fully execute ( more than a cup of hot beverage takes from inception to getting it into the belly 😀 ). I knew, that after a while this is going to happen, worked before with similar frameworks, but also already had some ideas in the back of my mind to fight it if it may become a problem. Obviously this was not a big irritation yet, but for a larger game it may become a certain source of frustration.

2016_07_17_Execution

The most simple and obvious solution was categorizing test-cases within a suite (UI, Options, Graphics, Tutorial etc…) and only execute immediately required categories. This took only a short time to develop and configure as NUnit already had support for it. I just had to put some extra properties here and there.
This was nice and worked well, taking less to test the crucial/modified parts faster, but of-course there was a much smarter idea there from where this one came from ( a full test still required 10+ minutes 🙂 , no way I could not improve on that 😉 )!

    
    
        
        
            Match Pause Restart
            Tests\ReplayMatchPauseRestart.xml
            
                
                Match
                Pause
            
        
        
    

2016_07_17_Categories_1  2016_07_17_Categories_2

So I investigated two common solutions to this problem (actually there is a third one which is manual: modify test-cases to make them simpler and shorter, but that is not a general solution, takes time and the gains are small), and I went with the “speeding up test-case execution” route. I haven’t find any good/common name for it, although it is a known solution, so I called it the “unlocked” game-loop. The concept is simple: when replaying the test-cases a different game-loop is used, which runs as fast as it can ( no vsync no sleep nothing like that, exercising the CPU/GPU like a mad man 😀 ), but the elapsed, total and accumulated time calculated and passed to the systems and game-objects of the game is mimicking the normal game-loop, so the game “believes” it is running at normal speed with the target 30 or 60 frames per second. I was certain that it is going to speed up execution, and at least cutting execution time in half with a simple game. I was wrong, it became much much faster 😀 . After the new game-loop, the full test set took not much more than 2 minutes instead of 11…

Take a look:


Note: the first two minutes show the normal execution of 6 test-cases and the rest of the video show the same tests executed with the “unlocked” game-loop.

The approach has some down-sides (as with every route in software development), e.g.: the game may use system time for certain features (although I think this should be avoided, since the game-loop provides an elapsed total time of the game execution handled the same way as the elapsed/accumulated time) and another one is full-screen/windowed mode toggling which is not supported by this feature at all (maybe in the future, I guess it could be done, just it would require some hacks, don’t know yet). For these problems I “cleverly” introduced a per-test-case setting to override the game-loop-unlocked configuration, so the execution speed-up can be disabled for “unstable” test-cases.

    
    true
    
        
        
            Toggle Fullscreen
            Tests\ReplayToggleFullscreen.xml
            
            false
        
        
    

Another “I’m not so happy about it” thing is, that it is a bit hackish and fully platform dependent solution currently, but I guess in time I will solve this problem 🙂 .

As I mentioned there was another route I could take to speed up test execution. I think it is a somewhat superior solution, but would have taken much more effort both software and hardware wise, so I decided to go with the simple one. NUnit has an open parallel execution engine add-on, and I think it requires no explanation why that route is superior, since the limiting factor would only be the number of machines I could harness, but setup (and stability?!) would be a much more complex issue. In time I may try it out, since I’m interested in the actual setup time it takes + I’m certain with a couple of boxes execution time would match the time it takes to run a unit test set 🙂 , but the current solution fully satisfies my needs and my work-flow.

The testing framework is in an extremely stable and usable state for production by now. I’m going to make good use of it for my current game too. In time I’m planning to add more features to it, so a “testing – part 4” entry may happen 🙂 , but not anytime soon + most probably I will only focus on smaller, usability enhancements and additions.

I’m still working on some framework-y code from time-to-time (maybe next entry will be similar, mostly technical) and the current game project is not yet ready for announcement, but with this game I’m going to do a more open development. So starting from the working prototype till the finished product, I’m going to post (weekly maybe?) releases with limited content to get feedback and improve on usability, balance and overall features from the first days and to reach more players interested in the game, even before going to greenlight/itch.io/hopefully-Steam etc…
Expect a somewhat playable version soon (I think within weeks).

Take care!

KREEP, post Steam.

Hi everyone!

Haven’t been around the INTERNETZ lately, sorry for that (been busy with a lot of work and important decisions to make 😐 ).
I promised a postmortem kind-a thingy after the Steam release of Operation KREEP, so here it goes!

Before I jump into numbers, I’m going to describe the project a little, what went well, what didn’t, the usual mortem content…
The Steam update and the release itself for the game was designed, developed and tracked as a project separate from the original game. This was done mainly due to being greenlit only a few months after the release. I already started working on new projects back than and the good news kind-of shocked me. Wasn’t expecting it and I didn’t want to rush it to the market, so I decided to take my time with the Steam release and prepare a “free” update for the game (note: I always referred to this as a free update, since all the original buyers received it on various stores for free…). While working on this update and the Steam release preparation, feature KREEP (pun intended 😀 ) and a smallish burnout (combined with a semi-serious health-issue) almost took the head of this project :(. It took a little longer than six months, but I finished the update and released my first Steam game on the 10th of June, 2016 🙂 !!!

The good:

I released a game on Steam! That alone is a great reward and a memorable experience!!!
Also while preparing for it, I could revisit a game I made and finished/released already and I could give a little more love and polish to it. In one way, that was really cool. I guess many probably had some bad feelings about finishing a project and still wanting to add some more to it here and there.
+ Steam integration was nice and surprisingly- easy/well documented :).

The bad:

Making the update was a CHORE. The actual procedure got pretty boring fast. I had an existing game of which I made the most out of I could on the first try, and now it did not feel special. I felt, that most of the modifications only added more to the game but did not make it any better (except for the input handling enhancements :), working on that feature was pretty cool!).

And the ugly:

I made the same preparations as with my previous two projects, written down and estimated tasks, and planned important milestones. I also tracked my progress while working on the update the same way, but this time, disciplined and organized work did not save me. I felt a rather big urge to make a better game out of Operation KREEP, because “it is going to be on Steam” :|, so a small feature-creep clouded my sight. Also I had semi-serious health problems with my stomach and duodenal, so I spent the last six months in glum mode :(.

Some numbers to certify these results/problems:
The original game took 430 hours to make, spanning over 6 months, where my original estimations and milestones predicted 300 work hours and somewhere between 4 to 5 months. Not the best probably, but certainly not bad results.
This update also took 6 months, but only required a tiny little less than 190 work hours. This mostly shows my rather bad work ethic regarding this project (and the bad mood 😦 ), since the estimation was even closer to the end result, than with the original game. I estimated around 150 work hours and around 3 months top to execute it! Based on my results with Operation KREEP, the update with the Steam release could have been done in three months with same amount of weekly work-time spent on the project, but I guess sometimes life just happens :(.

So, mortems have to have some conclusions, on how to do better next time, right?
I think, next time if I feel, that some tasks and features do not really add much to the game overall + I feel like I’m loosing interest in the project, I will re-evaluate my designs and plans, or I will start a tiny pet project parallel, so that I do not fully waste a lot of my gamedev time and efforts. Also I’m certain I will take the mentioned stuff into account regarding milestones.

Did it sell?!

I guess this interests many, so here goes nothing: it did not sell…
Before going any further, I have to mention, that for saying “it did not sell”, I’m taking into account, that it is a small and niche game + my marketing efforts took much less time and/or money, than half of the project costs, as it is the case with AAA titles :D. I wasn’t expecting it to sell thousands of units at all in the first place :). Nevertheless, even with this in mind, selling only a few units leaves a rather bad taste in my mouth :(.
The Steam release did not reach 100 units. Based on the wish-list additions, it may reach 500, probably if I drop the price later on, and that number sounds better, but I feel like the price was totally reasonable even for the original release. I guess it is common sense, but this is another example, that a game alone is not going to reach many people, not even on Steam. Real effort has to be poured into pr and marketing to actually get the word out and to find people interested in your game. I guess a more interesting game wouldn’t hurt either, but what can I say, I’m a sucker for retro and pixels :).

Still, I’m thankful for those sites and press people who tried the game, and even more thankful for those who wrote about it. Indie Retro News wrote a really cool review :), especially thankful for that!
+
For my next game, I’m going to work a lot more on the marketing front too, to reach a broader audience, who may enjoy it.

Future plans:

Recently I worked on many cool gamedev stuff and I’m prototyping my next game (not ready to show it yet, but it’s getting there).
+
I made a huge life-changing decision lately, which I’m going to keep as a “secret”, no official statement, because I suspect, I would get a lot of “Are you out of your mind?!” responses, and I don’t know how to handle that properly, but I guess it is not hard to figure it out. Here’s a hint:
Still here, writing about game development, the one and only job that keeps popping up in my life no matter where I go and what I do. A profession which I could spend a lifetime practicing, a profession which is intertwined with the person I am.

I tried many times to make a weekly habit out of writing these gamedev related post, but I failed miserably :(.
Now I’m going to have a lot of time to try it again, so expect my next post soon 🙂 😉 !
Best regards.