How fast can PerfCake be?

For a long time, we were wondering what are the limiting abilities of PerfCake. Mainly, what is the impact of PerfCake itself on the CPU. We created a simple scenario with the DefaultMessageGenerator and DummySender (that does nothing). We did not use validation for now as you can see in the scenario below. The aim of this exercise was to investigate thread management and proper data structures usage in the PerfCake core. On an Intel Xeon (2x 8 core CPU) we achieved over 75 billions of requests per second using 4096 concurrent threads/clients. Is your system that fast?

There is one more parameter to the test - the queue size. This is the size of an internal buffer used to prepare requests. If the queue was too short or too slow, it would block the maximal throughput. You can happily ignore this as the information is mainly for the developers. The colorful lines just make the charts look more joyful ;-)

We also wondered, how PerfCake can perform on a limited device based on an ARM CPU. Could you use your mobile phone as a load generator for example? So we run the same scenario on Cubieboard 2. The numbers are lower and the queue size now plays more important role. For our Fedora 19 installation on the device, there was a limit on the maximum number of threads for a system process. This is why the chart ends around 512 threads.

Following is the scenario.

Technical preview of DSL is in 3.3

As we promised, PerfCake 3.3 is out. The main reason for this release is a feature we originally planned for 3.0. We wanted to present this at GeeCON 2014 in Prague, so here is the first "technical preview". We also fixed some minor bugs - as we found them, there was no reason for postponing the fix. We hope to collect some feedback on the DSL implementation and possibly improve it.

We also look forward to see current and find some new users at GeeCON. Meanwhile, we also presented PerfCake in an IT company and discovered some new feature requests. Therefore, we started speaking about receivers, timers, correlators... You might possibly see some of the revolutionary concepts in release 4.0.

PerfCake 3.0 is out, so is 3.1 and 3.2...

For us, Release 3.0 is a major achievement and a significant success. We managed to fix multiple kids' diseases that were hidden in the code, we reworked problematic parts, automated many steps, revised the whole project infrastructure including CI, web site, release procedure... Especially the later one still needs some tweaking as explained later.
You can have a look on the list of closed issues for this release. The user experience should be now better. We prepared automated scenario migration to the last version (running this is however still little bit clumsy), we fixed all the issues with reporting (early or late results were not recorded), we did some PoC with Google charts output (and decided to solve that differently in version 4.0), we have some progress on the tooling (Eclipse plugin is almost done), we extended our tests to the level of advanced integration testing (running in-container testing of JMS sender for instance, special credit goes to an university student who implemented it), and many other changes that might be hardly visible from the outside but were important inside.
There is still one component in PerfCake that is not easy to extend or to implement on your own - the generators. They do not have an interface but an abstract parent class. This is something we would like to change for programmers' convenience.
From the users perspective, our priority is the tooling, documentation, migration from previous versions (and possibly migration from other tools), execution in a clustered environment, ability to directly represent results graphically (using charts).
We also have a plan to finally provide another way to specify the scenario - a DSL language. This feature is almost in a technical preview state.
So, what went wrong during the release and why we ended up with version 3.2? The reason is quite silly and it is a lesson learned for the next release. During the release procedure, we prepare everything on Github first. The master branch has a new version and the devel branch is set to continue on the next release. Then we prepare the artifacts and upload them to Sonatype for them to appear in the public Maven repository. Then our Continuous Integration environment automatically consumes the sources from master and runs all the tests. It also creates downloadable artifacts for automatic web pages refresh.
So after releasing version 3.0, we realized that the tests do not work well in the CI environment. We decided this must be caused by a different environment setup, implemented a fix, executed tests locally and released version 3.1.
But this did not help! Still the same errors appeared in the CI. After a more thorough inspection we realized that there is a problem with the javadoc Maven plugin (the recent version was backward incompatible). So we eventually fixed that and released version 3.2.
Next time, we must run our CI stuff before the upload to Sonatype.
And one more thing - there will be another release, version 3.3, shortly before GeeCON in Prague 2014. We want to present some cool new features there during the Lightning talks.

Release 2.1

The summer season is aproaching and we are half way on our journey to the next release of PerfCake that will be unleashed in Autumn. We are working hard to make PerfCake the #1 performance testing tool by adding new features, refactoring and optimizing the features that are already there and fixing bugs that eluded us and made it to the previous releases.

During midsummer we released hot-fix release v2.1 that fixes one such bug.

We would like to thank everybody who have helped... Thank you very much!

Release 2.0

Some time ago, we celebrated the Spring equinox with the 2nd release of PerfCake. Did we manage to implement everything we wanted? The answer is no again. Does it really matter? Not that much. We mainly focused on fixing some issues that the users were facing. Especially correct reporting of reaching 100% in time controlled runs. We also improved the framework performance so it has minimal influence on the system being measured. So the priorities have changed and we did some things that were not in the plan originally. It is a very nice feeling to see how the product gets matured. Like some good quality whisky...

Since release 1.0, we presented PerfCake to local public and we were surprised by the positive audience feedback. However, there are still questions we are not able to answer at the moment. Or more precisely, we are not able to provide the evidence for our answers. So before the next release, we would like to aim on being the best performance testing framework on the market. Because of that, we plan to publish comparisons with other frameworks and provide migration scenarios.

We learned that nice reporting is important to our users. So we would like to implement a reporter that generates charts.

For links to public talks, see our twitter.

PerfCake on LinuxAlt 2013

We are out and we are getting known!

You had the first opportunity to see a public demonstration of the basic features of our young and growing performance testing framework. Martin Večeřa and Jiří Sedláček performed the talk at the open-source software conference LinuxAlt 2013 held on the grounds of Brno University of Technology, Czech Republic on November 2nd, 2013.

Martin Večeřa mentioned our motivation to work in the performance testing area and the issues that can emerge in a process of the performance testing. Issues that PerfCake is trying to solve such as how to measure the performance, how to ensure the reliability of the measured metrics and what to take into account. Martin than briefly introduced the PerfCake architecture and the possibility to extend all the features PerfCake has.

Jiří Sedláček followed with the demonstration of a basic performance measurement done with PerfCake. Jiří showed how to get and install PerfCake and how to define and run a simple scenario to easily test performance of several HTTP endpoints including the reporting abilities.

Soon the record of the talk and the demonstration will be available online. The talk and the demonstration was performed in Czech but the English subtitles/transcript will be available soon as well.

We were excited to see how many people came to learn something about PerfCake. We would like to thank all the attendees especially for asking curious questions. It encouraged us to work even harder to achieve our goals such as to become the best tool on the market. It was all fun for us and we look forward to another opportunities to present PerfCake and to meet you, our growing community.

The upcoming release awaits your contributions!

Release One

So here we are. Our first release is out. For us, this was the first grown-up money-less open-source project we pushed so far. You know, sometimes you have a great idea but the lust goes away too soon. I do not know what makes PerfCake so different that it worked out. But I am really glad we made it. We learnt a lot, everybody played every role at least for a while (as there are more roles than available actors). And here we are!

Did we manage to achieve everything we wanted? Definitely not. Release 1.0 is not perfect. But we tried hard and we are almost there. The documentation needs more time to settle down, validation must be re-done, there is a design flaw in reporting the 100% status message for time based message generation... Are these just the details or major issues? This is what the users must tell us now.

At this point, I would like to thank to everybody who helped on the project. Not only developers and contributors, but those who reported or just found bugs, sent pull requests, provided their technical advice, motivated us to go on... Thank you very much!

Now, this is not the end. This is just the beginning and we must refresh our minds and head towards the second release, the Spring Edition!

Let's spend the winter writing some good piece of code. Onwards!