Testing Tips, Do's and Don'ts

Hey everyone! This post is to help give everyone an understanding about how we test internally so we can better help each other going forward.

Firstly, I'd like to point out a number of key elements we're looking at in these threads. Obviously we want to hear about any bugs or issues introduced by any new changes. We also want to hear general feedback and opinions. Most importantly when we're talking about DPS balance, we're looking for any unintended behaviors that are causing some to do more damage than expected.

To be blunt about it, we're way more worried about a power going live causing too much damage, than too little. When I hear that each and every power is "slightly underperforming" compared to the currently christened FotMs, that's fine to me. Absolutely without exception, once a power is released into the wild with a game update, new ways of playing that power quickly appear. And that power always begins performing better than tests on the test server showed. If a power does need further attention to increase damage, it's a delayed win. If a power needs further attention to reduce damage, the whole world lights on fire.

So with that said, in an effort to help us get more consistent results with one another here's a list of Testing Tips and Do's and Don'ts.

Character Setup

For AMs we're using:

• CR126 (live tests):
  • Precision Tester
    • Might: 8951
    • Precision: 4887
    • Wpn: 5806
    • TotalWpn: 10693
  • Might Tester
    • Might: 10423
    • Precision: 3380
    • Wpn: 5806
    • TotalWpn: 9186
  • Include RD mods, mainframe mods, generator mods, league proficiencies, and 160sp.
  • Targets:
  • • Short: 35500
    • Mid: 30000
    • Long: 25500

Testing Tips

There are a number of common pitfalls that can skew your results. Mostly these are due to differences in testing methodology, which this post will hopefully help address.

Integrity of Results

We're after accurate results. We're interested in what a power is capable of. That means we need to control for human error. The tester must be playing as best as humanly possible for the data to be acceptable.

Because of that, we use 10 second time slices for the Combat Parser interval. That way if you mess up, you lose at most 10 seconds of testing time. Here are two approaches you can take for handling when you flub your rotation while testing.

One approach is to simply throw out any 10 second time slice where you flubbed up the rotation. You do need to be careful about this though. Many rotations tend to hover in the 12 second range. This unfortunately spans beyond a 10 second slice. DPS fluctuates predictably over time. For example, if there's a predictable spike/dip pattern every other 10 second slice and mistakes cause you to throw out only dip results than your results will be more distorted than if you had thrown out an even number of spikes and dips.

The other approach you can take, which is the one I use, is to not make any mistakes ever... OK seriously though, I can usually go a few minutes without a making mistake. When I do make a mistake I stop the test and record the data collected to that point. Then I restart the test if I need more data. I wouldn't consider holding you to my fairly harsh standards but I do ask that you strive to only record and present reliable data.

Also keep in mind "bad data" means you literally flubbed up the rotation somehow. It doesn't mean you didn't crit enough. It doesn't mean that you crit too much. It doesn't mean you just don't like the number you're seeing. Obviously you're on your honor

How close should the results be and how much data do you need? Generally I'll get at least 30 good 10 second slices of data and see where it lands. We're happy when getting results that are about +/- 5% from what's expected.

Use the best conceivable power rotation and best weapon attacks for that range

Test using the three dummies in the league hall. This set of dummies can all be hit at one range without overlap. If you test versus the cluster of eight dummies on the left side, you may wind up hitting enemies from two different ranges at once which will give you incorrect results.

Test using the best conceivable power rotation and/or best weapon attack for that range. There are three ranges. They are:

• Short Range: 0 up to 7m
• Mid Range: 7m up to 13m
• Long Range: 13m up to max (25m)

Short range is easy. Just stand right next to the dummies. Long range is also fairly straightforward. Just back up until you're just within range. For mid-range, stand in the middle of the third square from the dummy.

When making comparisons, make sure you are comparing apples to apples. You don't want to compare a long range AM to a mid range or a short range for example.

If you're testing an AM that relies on supplemental weapon damage, make sure you are using a combo that makes sense for that range. If you're using a melee AM rotation but you're spamming WM DW Explosive Shot you're not maximizing your damage because you're using a ranged weapon attack with a melee AM.

In fact, we don't use WM when testing AM. You don't have too. Every weapon has a combo that matches the damage output of WM when used with the correct ability strategy. Here are those:

Weapon: Combo, ability rotation strategy

• Brawling: Haymaker, clipping
• Bow: Arrow Storm, optimal refresh (AS damage is broken skip testing with bow for now)
• Dual Pistol: Full-Auto (3-tick-switch w/ Sweep Shot), optimal refresh
• Dual Wield: 4-6x Dual Flurry, clipping
• Hand Blaster: Scissor Kick, clipping
• Martial Arts: Smoke Bomb, clipping
• One Handed: Flip Slash, clipping
• Rifle: Mortar, optimal refresh
• Shield: Handspring kick, clipping
• Staff: Overhead Strike or Downward Smash, clipping
• Two Handed: Doom Spin, optimal refresh

Finding a Good Rotation
Generally most powers have one or more sources of parallel damage that can be rolling while the main damage rotation takes place. These are often things like fields, auras and weapon buffs. You want to use these if they actually increase your damage output. Sometime you can use more than one of these and it will further improve your damage but not always.

Sometimes the time spent activating the ability actually reduces your overall DPS. There is a large amount of variability between the powers in this regard. So be certain that every ability activation is actually increasing your damage output.

Thanks for Your Help!

We appreciate everyone who takes the time to help us test and give feedback about what's on PC Test. We hope you find these insightful. We look forward to continued to collaboration!