tag:blogger.com,1999:blog-85324671280770168782024-03-12T18:57:13.217-07:00PerfCake Performance Testing FrameworkA Lightweight Performance Testing FrameworkMartin Večeřahttp://www.blogger.com/profile/01257832382088091783noreply@blogger.comBlogger15125tag:blogger.com,1999:blog-8532467128077016878.post-78074781468600077672016-09-21T12:23:00.000-07:002016-09-22T00:01:57.345-07:00Release 7.0 and The Vision<p>I would like to officially welcome you to 2016 autumn/fall (depending where you are). The days are getting shorter and the weather is becoming more moody. Fortunately, there are even good and exciting news. The number 7 in the PerfCake version number is not only lucky. It also introduces 57 total enhancements and 17 bug fixes. We spent extraordinary efforts on bringing the best performance testing tool we could to you. Feel free to <strong><a href="https://www.perfcake.org/download/">download now!</a></strong></p>
<p>Except for code enhancements, we also completed our guides (<a href="https://www.perfcake.org/guide/user/">Users' Guide</a> and <a href="https://www.perfcake.org/guide/devel/">Developers' Guide</a>) to carry the best up-to-date information. In addition to this, we realized that, first, performance testing is not an easy job and there aren't too many resources on how to do that properly, and second, some scenarios in PerfCake that can be especially powerful aren't very obvious on first sight. To overcome these difficulties, we started writing two books on LeanPub. First is <a href="https://leanpub.com/perfengineering">Performance Engineering</a> and the second is <a href="https://leanpub.com/perfcake">PerfCake: Performance Testing Scenarios</a>. The first one will ask you to contribute a small amount of money to support our work, the second one is and will be for free forever, however, small contributions will be gratefully appreciated.</p>
<p>I would like to use this opportunity to also introduce you to our vision of future of performance testing and what we do to make it happen. For a long time, we know that we suffered in the area of development of performance test scenarios for PerfCake. No matter how powerful the XML is, we need something more visually appealing. Currently, an IDE plugin is under development that will bring you not just a scenario editor but also a debugging tool. You will be able to use the new debugging facility in PerfCake and monitor an actual test execution.</p>
<p>Regarding runtime, we would like to work more in a way of performance testing as a service. Users could use a Web UI to connect their favorite cloud environment (e.g. EC2, Azul, GCE, OpenStack...) and use the machines in the cloud as agents to run the performance scenario in a cluster determining geographical locations and other attributes.</p>
<p>The results would be then stored in a PerfRepo which is already a part of our portfolio. The PerfRepo is undergoing a rework to cover more common use cases in fewer clicks and be more user friendly. Also the integration between PerfCake and PerfRepo will be more seamless and automated. The results will be stored for later examination, comparison between various runs, and reporting. Again as part of our Web UI suite.</p>
<p>Regarding the PerfCake core, we are not aiming at any "shake the world" changes for spring release. Unless there is any demand on specific features, we will cover <a href="https://github.com/PerfCake/PerfCake/issues">the issues currently open on Github</a> and concentrate more on the books, documentation, video blogs and viral marketing in general. The development efforts will be spent more on the supporting tooling.</p>
<p>Just for curiosity, the code name of this release is <em>Raspberry Moon</em> according to a full moon we had a chance to see while thinking about the code name. It was a beautiful full moon low on the horizon and it had a nice red-pink (like raspberries) color.</p>Martin Večeřahttp://www.blogger.com/profile/01257832382088091783noreply@blogger.com0tag:blogger.com,1999:blog-8532467128077016878.post-38738056470904196302015-12-14T00:21:00.002-08:002015-12-14T01:00:55.737-08:00PerfRepo in DockerUsing <a href="https://github.com/PerfCake/PerfRepo">PerfRepo</a> now became much easier than it was before. How so? It comes to <a href="https://www.docker.com/">Docker</a>. To run an instance of PerfRepo is as simple as starting 2 docker containers - one with PerfRepo web application and the second with the DB - and link them together.<br />
<br />
So, let's have a look at how to run PerfRepo with Docker...<br />
<ul>
<li>Start a DB using <b>perfcake/perfrepo-postgresql</b> image with container name <b>perfrepo-db</b>:</li>
</ul>
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">$ docker run -d --name perfrepo-db perfcake/perfrepo-postgresql:v1.4</pre>
<br />
<ul>
<li>Start PerfRepo container from <b>perfcake/perfrepo</b> image at a specific port (e.g. 5000) and link it with the previously created DB container (the link alias needs to be <b>perfrepo-db</b>):</li>
</ul>
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">$ docker run -d -p 5000:8080 --name perfrepo --link perfrepo-db perfcake/perfrepo:v1.4</pre>
<ul>
<li>Verify Docker containers are running:</li>
</ul>
<pre style="background-color: #f7f7f7; border-radius: 3px; box-sizing: border-box; color: #333333; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 13.6px; font-stretch: normal; line-height: 1.45; overflow: auto; padding: 16px; word-break: normal; word-wrap: normal;">$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
251863199f81 perfcake/perfrepo:v1.4 <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/root/jboss-eap-6.4/<span class="pl-pds" style="box-sizing: border-box;">"</span></span> 4 seconds ago Up 3 seconds 0.0.0.0:5000-<span class="pl-k" style="box-sizing: border-box; color: #a71d5d;">></span>8080/tcp perfrepo
baaeacfd98f5 perfcake/perfrepo-postgresql:v1.4 <span class="pl-s" style="box-sizing: border-box; color: #183691;"><span class="pl-pds" style="box-sizing: border-box;">"</span>/usr/bin/postgres -D<span class="pl-pds" style="box-sizing: border-box;">"</span></span> 35 minutes ago Up 35 minutes 5432/tcp perfrepo-db</pre>
<div>
<br /></div>
PerfRepo should be running on the port set in the previous step (e.g. 5000) <a href="http://localhost:5000/">http://localhost:5000</a>. The default user and password are:<br />
<ul>
<li>login: <b>perfrepouser</b></li>
<li>password: <b>perfrepouser1.</b></li>
</ul>
And that's all it takes...<br />
<div>
<br />
<div>
The docker images are ready for you in the <a href="https://hub.docker.com/search/?q=perfcake/perfrepo">Docker Hub</a> so have fun!!</div>
</div>
Pavel Macíkhttp://www.blogger.com/profile/16312339465957297540noreply@blogger.com0tag:blogger.com,1999:blog-8532467128077016878.post-41882012015832828752015-06-25T22:47:00.002-07:002015-06-26T04:14:33.056-07:00CzechTest 2015This time, we would like to share our talk with you in advance. Let us introduce you the planned <a href="http://www.czechtest.com/perfcake-performance-testing-made-piece-cake" target="_blank">Friday talk</a> at <a href="http://www.czechtest.com/" target="_blank">CzechTest</a>, the presentation and the description of the demo. You won't be able to have the unique experience the demo will bring to live participants. We will be sharing recording later of course. But you know, you won't be able to say "I was there".<br />
You can find the presentation at <a href="https://goo.gl/RrYrMc" target="_blank">Google Drive already</a>.<br />
Let me start with a short live questionnaire.<br />
<ol>
<li>Have you ever considered performance of the software you were developing?</li>
<li>Have you ever used any performance measurement tool?</li>
<li>Which one?</li>
<li>How was your experience with the tool you have used?</li>
<li>Have you considered the performance of the tool itself?</li>
</ol>
What we have discovered over the time is that most people use <a href="http://jmeter.apache.org/" target="_blank">jMeter</a>. Which is a fact we do not understand at all. It is not easy to use, it has cumbersome user interface with ugly design, it is not very user friendly. Plus it does not have that great performance at all.<br />
Why didn't you consider <a href="https://www.perfcake.org/" target="_blank">PerfCake</a>?<br />
It is the first open-source performance measurement... well, I can even say the first performance measurement tool ever created with all the critical aspects in mind. We take care of user experience, ease of use, the minimal footprint and great performance. It can be used as a load generator, to run stress tests (finding the maximal application throughput), to run soak tests (endurance testing), to validate application responses under stress, to generate nice chart report and many more.<br />
All of this can be done very fast. With the highest performance among such tools. We have measured this.<br />
From our experience, performance testing is very complex area. Developing a single performance measurement tool that fits all is like trying to sew trousers that fits all. Impossible. If you do not want to make them extra large. But this is not fitting, this is clumsy.<br />
Many tiny details need to be incorporated for individual requirements. Instead of trying to make everyone happy we decided to use a different concept. We do not want to develop a user interface that can do all but nothing because nobody understands it.<br />
Instead, we have developed a simple, stable and reliable platform with well documented interfaces that can be easily extended. <br />
In the end of the day, just people with developers' skills run performance tests. So why we don't let them develop what they need. They know better. You know better!<br />
<br />
We owe you, our happy users, a complex Developers' guide at the moment. We promise to finish it for release 5.0. Should you need any help, you could use <a href="https://www.perfcake.org/support/" target="_blank">the contact form</a> at our web pages.<br />
Let's have a look at few examples. These should demonstrate in deeds why we believe we are presenting you the best performance measurement tool ever created.<br />
First, we will show you how to easily develop your first performance test in a nice user interface in <a href="https://www.jetbrains.com/idea/" target="_blank">IntelliJ Idea</a>. So to give you some eye candy.<br />
Next, for more advanced guys, we will show you how to get latest updates on the code before the 5.0 release. And we will show you the ease of use from the command line. The command line is nothing to be afraid of, especially with <a href="https://www.perfcake.org/" target="_blank">PerfCake</a>.<br />
Having fallen in love with the command line, we will conclude with developing a slightly more complicated scenario using our cool almost-spoken-like DSL language. We can verify several HTTP echo implementations in your favorite languages. Finally, we will have a look at a more complex scenario with sequences and validators.<br />
<br />
The <a href="https://github.com/PerfCake/Demos/tree/master/CzechTest2015" target="_blank">examples can be found at GitHub</a>.Martin Večeřahttp://www.blogger.com/profile/01257832382088091783noreply@blogger.com0tag:blogger.com,1999:blog-8532467128077016878.post-80902904230291502732015-04-27T11:54:00.001-07:002015-04-27T11:54:52.227-07:00University release 4.1We have been invited to teach a lecture on performance testing at <a href="http://www.fi.muni.cz/" target="_blank">Faculty of Informatics, Masaryk University</a>. The choice of the performance testing tool to use during the practical part was clear (PerfCake if it wasn't obvious). We already knew about a couple of things that were broken in 4.0. So for the <a href="https://github.com/PerfCake/Demos/tree/master/FiMuni2015" target="_blank">demo</a> to be as smooth as possible we have decided to create a dedicated University release, numbered 4.1.<br />
It has <a href="https://github.com/PerfCake/PerfCake/issues?q=milestone%3A%22Release+4.1%22" target="_blank">five bug fixes, three enhancements</a> and we also updated our web pages. All our friends and users are strongly recommended to update to version 4.1 to avoid any possible inconveniences.<br />
Did the lecture went all that smooth in the end? Of course not. What was the deal breaker? The students were supposed to run PerfCake on a commercial operating system by Microsoft. This has not been tested thoroughly by us. We found out that the main script in the binary release works only when executed from the <a href="https://github.com/PerfCake/PerfCake/issues/204" target="_blank">main installation directory</a>. Plus some more minor things that we were able to overcome. Some of them are already fixed in the <a href="https://github.com/PerfCake/PerfCake/tree/devel" target="_blank">devel</a> branch but we are unlikely to create another release now. Possibly later in the summer as we are still hoping to be able to show PerfCake in public again.Martin Večeřahttp://www.blogger.com/profile/01257832382088091783noreply@blogger.com0tag:blogger.com,1999:blog-8532467128077016878.post-77199156444471793492015-04-19T13:25:00.002-07:002015-04-19T13:25:39.536-07:00PerfCake has a cool new logoAs we already <a href="https://twitter.com/PerfCakeTool/status/589089993911590912" target="_blank">announced on Twitter</a>, PerfCake has a <a href="https://github.com/PerfCake/Web/blob/master/images/logo-new.png" target="_blank">cool new logo</a>! It is also available in <a href="https://github.com/PerfCake/Web/blob/master/images/logos2.svg" target="_blank">SVG format</a> with its variants.<br />
The previous logo we used to use was fine. We needed something for a beginning project and <a href="https://github.com/pmacik" target="_blank">Pavel</a> created the original one quickly. Btw. did you know that it depicted an inner side of a cake cut from a close lookup? :-)<br />
As PerfCake matured, we knew that we are going to need something that resonates more with us. We have a stable core so it really deserved something nice. We also tweaked our web page a little bit. But more changes are likely to come to make it even more simple.<br />
We are especially excited about the new logo because it enables us to further develop the brand. We created a nice presentation template already. Come to a public event to see it in action! We can now create a boxed version of PerfCake, we can create cool stuff like t-shirts... The possibilities are now unlimited!<br />
Any feedback on the new logo is highly appreciated!Martin Večeřahttp://www.blogger.com/profile/01257832382088091783noreply@blogger.com0tag:blogger.com,1999:blog-8532467128077016878.post-82970055242864080502015-03-23T12:30:00.000-07:002015-04-17T09:03:21.989-07:00From 4.0 to 4.1As usually, we celebrated spring equinox with an even numbered release of PerfCake. We managed to release a couple of days earlier. The main reason for that was that we had a planned vacation for the release period.<br />
This release has some really <a href="https://github.com/PerfCake/PerfCake/blob/v4.0/CHANGES.md" target="_blank">cool features</a> that makes PerfCake even easier for usage. Actually there are 26 of enhancements and new features in this release. Just have a look on the list to see if you could make a use of them.<br />
Except for these cool features there is one thing we are especially proud of in this release. This is its stability. We finally managed to get rid of all the raise conditions appearing now and then and bothering us constantly. I do not want to claim that there aren't any now. But there aren't any we are aware of. And there is a fair chance that nobody will discover a new one (until we change the code ;-)).<br />
Unfortunately, bad things happen and there will be version 4.1 soon. Especially due to <a href="https://github.com/PerfCake/PerfCake/issues/192" target="_blank">a bug that prevents smooth shell script usage randomly</a>.<br />
We also decided to fix several other issues. This is mainly because we want to get ready for a possibility to present PerfCake publicly soon again. All information on that will be shared once we have an official confirmation. Stay tuned!Martin Večeřahttp://www.blogger.com/profile/01257832382088091783noreply@blogger.com0tag:blogger.com,1999:blog-8532467128077016878.post-11684822812183121632015-02-08T13:39:00.000-08:002015-04-13T12:32:08.078-07:00DevConf 2015This year, we were accepted with our talk at <a href="http://www.devconf.cz/" target="_blank">DevConf</a> as well. We always try to show something new. So we worked hard to be able to show major cool features.<br />
The first is the ability to automatically generate chart reports using <a href="https://developers.google.com/chart/" target="_blank">Google Charts</a>. This is provided in the form of a <a href="https://github.com/PerfCake/PerfCake/blob/v4.0/src/main/java/org/perfcake/reporting/destinations/ChartDestination.java" target="_blank"><span style="font-family: "Courier New",Courier,monospace;">ChartDestination</span></a>. PerfCake also automatically combines recent results with any previously existing in the output directory which gives you an automatic comparison.<br />
We have demonstrated this with several HTTP Reverse Echo clients, each of which was implemented in a different language. We reused the same scenario to show the user friendliness of PerfCake and the fact that PerfCake does not care about the language the measured application is developed in.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-xl6v8vdAEM7ZcUq3x0vZsYMRQGmiodrllGKzP5IQ1VgMLEFshgjjTqMt__bWOlObKtFm5ft3C_DbFu0nGvG79fEC-yqpdOIquKHQOwsBVPIRC2OX7viBvh3cY5r7BLY_q9COSlJdGYU/s1600/perfcake-chart-dest.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-xl6v8vdAEM7ZcUq3x0vZsYMRQGmiodrllGKzP5IQ1VgMLEFshgjjTqMt__bWOlObKtFm5ft3C_DbFu0nGvG79fEC-yqpdOIquKHQOwsBVPIRC2OX7viBvh3cY5r7BLY_q9COSlJdGYU/s1600/perfcake-chart-dest.png" height="158" width="400" /></a></div>
<br />
The second major feature was automatic memory leak detection. This works only for Java currently, however, we were able to detect all the leaks.<br />
The way we do presentations is that we try to evolve over time. We try hard to create something new for anybody visiting all of our presentation still have some fun. The thing is that we did not see many people coming back and listening to new things. After this last presentation we are likely to reevaluate our approach and make sure we always presents the basics. If you could provide your opinions on that, we would love to take them into account. <br />
<ul>
<li><a href="https://youtu.be/5-Rtea-lJzU?t=3m07s" target="_blank">Presentation video</a></li>
<li><a href="https://github.com/PerfCake/Demos/tree/master/DevConf2015" target="_blank">Demo resources</a></li>
</ul>
<br />Martin Večeřahttp://www.blogger.com/profile/01257832382088091783noreply@blogger.com0tag:blogger.com,1999:blog-8532467128077016878.post-57566475181985835732014-10-21T10:49:00.000-07:002014-10-22T08:26:44.022-07:00How fast can PerfCake be?<p>For a long time, we were wondering what are the limiting abilities of PerfCake. Mainly, what is the impact of PerfCake itself on the CPU. We created a simple scenario with the <i>DefaultMessageGenerator</i> and <i>DummySender</i> (that does nothing). We did not use validation for now as you can see in the scenario below. The aim of this exercise was to investigate thread management and proper data structures usage in the PerfCake core. On an Intel Xeon (2x 8 core CPU) we achieved over 75 billions of requests per second using 4096 concurrent threads/clients. Is your system that fast?</p>
<p>There is one more parameter to the test - the queue size. This is the size of an internal buffer used to prepare requests. If the queue was too short or too slow, it would block the maximal throughput. You can happily ignore this as the information is mainly for the developers. The colorful lines just make the charts look more joyful ;-)</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEThy96hWVx-R2OBQN0CwVpG9KFk_82dDqRL_PA5xU_5cAToLceRG-fvd_pWGACprUyvWGSdWL0a9gm5rrw-rWGw5n-QNtPMqU-5j0ASomn8xd-vQR2LeNvDJhBZ10p17CUKjFtYC7qoo/s1600/throughput-xeon.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEThy96hWVx-R2OBQN0CwVpG9KFk_82dDqRL_PA5xU_5cAToLceRG-fvd_pWGACprUyvWGSdWL0a9gm5rrw-rWGw5n-QNtPMqU-5j0ASomn8xd-vQR2LeNvDJhBZ10p17CUKjFtYC7qoo/s400/throughput-xeon.png" /></a></div>
<p>We also wondered, how PerfCake can perform on a limited device based on an ARM CPU. Could you use your mobile phone as a load generator for example? So we run the same scenario on <a href="http://cubieboard.org/2013/06/19/cubieboard2-is-here/">Cubieboard 2</a>. The numbers are lower and the queue size now plays more important role. For our Fedora 19 installation on the device, there was a limit on the maximum number of threads for a system process. This is why the chart ends around 512 threads.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjiEPvDmVQD1wo51IIDBv-JLovMYks8744N8zVEfsCf4N8yd0nnbo_YKviCz4tN8wMFbL3PKep4TCSGWg4MGKPcW7lnxfafkRtHNkPfHdbx-mbBPjQ8DAiHMaJkwrz3s08V72OgZVTk8CY/s1600/throughput-cubieboard2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjiEPvDmVQD1wo51IIDBv-JLovMYks8744N8zVEfsCf4N8yd0nnbo_YKviCz4tN8wMFbL3PKep4TCSGWg4MGKPcW7lnxfafkRtHNkPfHdbx-mbBPjQ8DAiHMaJkwrz3s08V72OgZVTk8CY/s400/throughput-cubieboard2.png" /></a></div>
<p>Following is the scenario.</p>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2msebtMnahFUL7hwfOnCte4ozB6QvTN8e1xNmUtBsruGAycLIuqKMpn9WOZOlPJoYwcfDvqVvBz5bjHDv4Zj0DjS7kDA5edm6W2UtXCN396FfaJKmCDkV-NrMY13j63KQZvNBGUF0Y_E/s1600/scenario-dummy.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2msebtMnahFUL7hwfOnCte4ozB6QvTN8e1xNmUtBsruGAycLIuqKMpn9WOZOlPJoYwcfDvqVvBz5bjHDv4Zj0DjS7kDA5edm6W2UtXCN396FfaJKmCDkV-NrMY13j63KQZvNBGUF0Y_E/s400/scenario-dummy.png" /></a></div>Martin Večeřahttp://www.blogger.com/profile/01257832382088091783noreply@blogger.com0tag:blogger.com,1999:blog-8532467128077016878.post-53390110925695476992014-10-21T10:14:00.002-07:002014-10-21T10:14:29.193-07:00Technical preview of DSL is in 3.3<p>As we promised, <a href="http://search.maven.org/#artifactdetails|org.perfcake|perfcake|3.3|jar">PerfCake 3.3 is out</a>. The main reason for this release is <a href="https://github.com/PerfCake/PerfCake/issues/94">a feature</a> we originally planned for 3.0. We wanted to present this at <a href="http://www.geecon.cz/">GeeCON 2014 in Prague</a>, so here is the first "technical preview". We also fixed some minor bugs - as we found them, there was no reason for postponing the fix. We hope to collect some feedback on the DSL implementation and possibly improve it.</p>
<p>We also look forward to see current and find some new users at <a href="http://www.geecon.cz/">GeeCON</a>. Meanwhile, we also presented PerfCake in an IT company and discovered some new feature requests. Therefore, we started speaking about <i>receivers, timers, correlators...</i> You might possibly see some of the revolutionary concepts in release 4.0.</p>
Martin Večeřahttp://www.blogger.com/profile/01257832382088091783noreply@blogger.com0tag:blogger.com,1999:blog-8532467128077016878.post-17467296229379513452014-10-10T08:25:00.001-07:002014-10-10T13:02:00.542-07:00PerfCake 3.0 is out, so is 3.1 and 3.2...For us, Release 3.0 is a major achievement and a significant success. We managed to fix multiple kids' diseases that were hidden in the code, we reworked problematic parts, automated many steps, revised the whole project infrastructure including CI, web site, release procedure... Especially the later one still needs some tweaking as explained later.<br />
You can have a look on the list of <a href="https://github.com/PerfCake/PerfCake/issues?q=milestone%3A%22Release+3.0%22" target="_blank">closed issues for this release</a>. The user experience should be now better. We prepared automated scenario migration to the last version (running this is however still little bit clumsy), we fixed all the issues with reporting (early or late results were not recorded), we did some PoC with Google charts output (and decided to solve that differently in version 4.0), we have some progress on the tooling (Eclipse plugin is almost done), we extended our tests to the level of advanced integration testing (running in-container testing of JMS sender for instance, special credit goes to an university student who implemented it), and many other changes that might be hardly visible from the outside but were important inside.<br />
There is still one component in PerfCake that is not easy to extend or to implement on your own - the generators. They do not have an interface but an abstract parent class. This is something we would like to change for programmers' convenience.<br />
From the users perspective, our priority is the tooling, documentation, migration from previous versions (and possibly migration from other tools), execution in a clustered environment, ability to directly represent results graphically (using charts).<br />
We also have a plan to finally provide another way to specify the scenario - <a href="https://github.com/PerfCake/PerfCake/issues/94" target="_blank">a DSL language</a>. This feature is almost in a technical preview state.<br />
So, what went wrong during the release and why we ended up with version 3.2? The reason is quite silly and it is a lesson learned for the next release. During the release procedure, we prepare everything on Github first. The master branch has a new version and the devel branch is set to continue on the next release. Then we prepare the artifacts and upload them to <a href="https://oss.sonatype.org/" target="_blank">Sonatype</a> for them to appear in the <a href="http://search.maven.org/" target="_blank">public Maven repository</a>. Then our <a href="https://perfcake.ci.cloudbees.com/" target="_blank">Continuous Integration environment</a> automatically consumes the sources from master and runs all the tests. It also creates downloadable artifacts for automatic <a href="https://www.perfcake.org/" target="_blank">web pages</a> refresh.<br />
So after releasing version 3.0, we realized that the tests do not work well in the <a href="https://perfcake.ci.cloudbees.com/" target="_blank">CI environment</a>. We decided this must be caused by a different environment setup, implemented a fix, executed tests locally and released version 3.1.<br />
But this did not help! Still the same errors appeared in the <a href="https://perfcake.ci.cloudbees.com/" target="_blank">CI</a>. After a more thorough inspection we realized that there is a problem with the <a href="http://maven.apache.org/plugins/maven-javadoc-plugin/" target="_blank">javadoc Maven plugin</a> (the recent version was backward incompatible). So we eventually fixed that and released version 3.2.<br />
Next time, we must run our <a href="https://perfcake.ci.cloudbees.com/" target="_blank">CI</a> stuff before the upload to <a href="https://oss.sonatype.org/" target="_blank">Sonatype</a>.<br />
And one more thing - there will be another release, version 3.3, shortly before <a href="http://2014.geecon.cz/" target="_blank">GeeCON in Prague 2014</a>. We want to present some cool new features there during the Lightning talks.Martin Večeřahttp://www.blogger.com/profile/01257832382088091783noreply@blogger.com0tag:blogger.com,1999:blog-8532467128077016878.post-49454273558369254202014-06-21T03:53:00.000-07:002014-06-21T03:59:49.263-07:00Release 2.1<span style="line-height: 18px; white-space: pre-wrap;">The summer season is aproaching and we are half way on our journey to the <a href="https://github.com/PerfCake/PerfCake"><span style="color: #6fa8dc;">next release</span></a> of PerfCake that will be unleashed in Autumn. We are working hard to make PerfCake the #1 performance testing tool by adding new features, refactoring and optimizing the features that are already there and fixing bugs that eluded us and made it to the previous releases.</span><br />
<br />
<span style="line-height: 18px; white-space: pre-wrap;">During midsummer we released <a href="https://www.perfcake.org/download/"><span style="color: #6fa8dc;">hot-fix release v2.1</span></a> that fixes one such <a href="https://github.com/PerfCake/PerfCake/issues/119"><span style="color: #6fa8dc;">bug</span></a>.</span><br />
<br />
<span style="line-height: 18px; white-space: pre-wrap;">We would like to thank everybody who have helped... Thank you very much!</span>Pavel Macíkhttp://www.blogger.com/profile/16312339465957297540noreply@blogger.com0tag:blogger.com,1999:blog-8532467128077016878.post-75121519503212701212014-04-09T09:40:00.000-07:002014-04-09T09:40:27.404-07:00Release 2.0<p>Some time ago, we celebrated the Spring equinox with the <a href="https://www.perfcake.org/download/" target="_blank">2nd release of PerfCake</a>. Did we manage to implement everything we wanted? The answer is <i>no</i> again. Does it really matter? Not that much. We mainly focused on fixing some issues that the users were facing. Especially correct reporting of reaching 100% in time controlled runs. We also improved the framework performance so it has minimal influence on the system being measured. So the priorities have changed and we did some things that were not in the plan originally. It is a very nice feeling to see how the product gets matured. Like some good quality whisky...</p><p>
Since release 1.0, we presented PerfCake to local public and we were surprised by the positive audience feedback. However, there are still questions we are not able to answer at the moment. Or more precisely, we are not able to provide the evidence for our answers. So before the next release, we would like to aim on being the best performance testing framework on the market. Because of that, we plan to publish comparisons with other frameworks and provide migration scenarios.</p><p>We learned that nice reporting is important to our users. So we would like to implement a reporter that generates charts.</p><p>
For links to public talks, see <a href="https://twitter.com/PerfCakeTool" target="_blank">our twitter</a>.</p>Martin Večeřahttp://www.blogger.com/profile/01257832382088091783noreply@blogger.com0tag:blogger.com,1999:blog-8532467128077016878.post-71863425989566853252013-11-26T06:56:00.000-08:002014-04-10T04:36:33.789-07:00PerfCake on LinuxAlt 2013<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="line-height: 18px; white-space: pre-wrap;">We are out and we are getting known!</span><br />
<span style="line-height: 18px; white-space: pre-wrap;"><br /></span>
<span style="line-height: 18px; white-space: pre-wrap;">You had the first opportunity to see a <a href="http://goo.gl/cUf2oh"><span style="background-color: white; color: #6fa8dc;">public demonstration</span></a> of the basic features of our young and growing performance testing framework. Martin Večeřa and Jiří Sedláček performed the talk at the open-source software conference <a href="http://www.linuxalt.cz/2013"><span style="color: #6fa8dc;">LinuxAlt 2013</span></a> held on the grounds of Brno University of Technology, Czech Republic on November 2nd, 2013.</span><br />
<span style="line-height: 18px; white-space: pre-wrap;"><br /></span>
<span style="line-height: 18px; white-space: pre-wrap;">Martin Večeřa mentioned our motivation to work in the performance testing area and the issues that can emerge in a process of the performance testing. Issues that PerfCake is trying to solve such as how to measure the performance, how to ensure the reliability of the measured metrics and what to take into account. Martin than briefly introduced the PerfCake architecture and the possibility to extend all the features PerfCake has.</span><br />
<span style="line-height: 18px; white-space: pre-wrap;"><br /></span>
<span style="line-height: 18px; white-space: pre-wrap;">Jiří Sedláček followed with the demonstration of a basic performance measurement done with PerfCake. Jiří showed how to get and install PerfCake and how to define and run a simple scenario to easily test performance of several HTTP endpoints including the reporting abilities.</span><br />
<span style="line-height: 18px; white-space: pre-wrap;"><br /></span>
<span style="line-height: 18px; white-space: pre-wrap;">Soon the record of the talk and the demonstration will be available online. The talk and the demonstration was performed in Czech but the English subtitles/transcript will be available soon as well.</span><br />
<span style="line-height: 18px; white-space: pre-wrap;"><br /></span>
<span style="line-height: 18px; white-space: pre-wrap;">We were excited to see how many people came to learn something about PerfCake. We would like to thank all the attendees especially for asking curious questions. It encouraged us to work even harder to achieve our goals such as <i>to become the best tool on the market</i>. It was all fun for us and we look forward to another opportunities to present PerfCake and to meet you, our growing community.</span><br />
<span style="line-height: 18px; white-space: pre-wrap;"><br /></span>
<span style="line-height: 18px; white-space: pre-wrap;">The upcoming release awaits your contributions!</span><br />
<div style="line-height: 1.15;">
<br /></div>
</div>
<div lang="en-US" style="margin-bottom: 0cm;">
<span style="font-size: 16px; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span></div>
Pavel Macíkhttp://www.blogger.com/profile/16312339465957297540noreply@blogger.com0tag:blogger.com,1999:blog-8532467128077016878.post-72087779115874703352013-09-29T10:36:00.003-07:002014-04-09T09:40:43.653-07:00Release One<p>So here we are. Our first release is out. For us, this was the first grown-up money-less open-source project we pushed so far. You know, sometimes you have a great idea but the lust goes away too soon. I do not know what makes PerfCake so different that it worked out. But I am really glad we made it. We learnt a lot, everybody played every role at least for a while (as there are more roles than available actors). And here we are!</p>
<p>Did we manage to achieve everything we wanted? Definitely not. Release 1.0 is not perfect. But we tried hard and we are almost there. The documentation needs more time to settle down, validation must be re-done, there is a design flaw in reporting the 100% status message for time based message generation... Are these just the details or major issues? This is what the users must tell us now.</p>
<p>At this point, I would like to thank to everybody who helped on the project. Not only developers and contributors, but those who reported or just found bugs, sent pull requests, provided their technical advice, motivated us to go on... Thank you very much!</p>
<p>Now, this is not the end. This is just the beginning and we must refresh our minds and head towards the second release, the Spring Edition!</p>
<p>Let's spend the winter writing some good piece of code. Onwards!</p>Martin Večeřahttp://www.blogger.com/profile/01257832382088091783noreply@blogger.com0tag:blogger.com,1999:blog-8532467128077016878.post-46679761753220603242013-07-23T12:14:00.000-07:002013-09-18T14:45:04.327-07:00Welcome to PerfCake<p>
Today I received a printed copy of a new book published at Pack - <a href="http://www.packtpub.com/jboss-as-7-development/book">JBoss AS 7 Development</a>. I did a review of the book (I appologize to Francesco Marchioni for being such a bas*ard) so they gave me a small paragraph in the credits. How does that relate to PerfCake? There is a hardly noticeable marketing of PerfCake. Well, to be honest, it says that I publish articles on the PerfCake blog. So I guess it is the right time to start now ;-)
</p>
<p>
<strong>So what is PerfCake?</strong> As you can see on <a href="http://www.perfcake.org">its home page</a> - <em>PerfCake is a lightweight performance testing tool and a load generator with the aim to be minimalistic, easy to use, provide stable results, have minimum influence on the measured system, be platform independent, use component design, allow high troughput.</em>
There is also a more detailed description...
</p>
<p>
<strong>So what really is PerfCake?</strong> Ok, did you play with some nice <a href="http://www.lego.com/">building kit®</a> as a kid? We did! And we liked it so much that we wanted exactly the same kit for performance testing. For us to be able to independently specify what, how and when should be tested. Once we might want to compose a castle, the other time a space ship, and yet we can reuse the basic blocks. Why did not we choose any existing tool? There is a separate blog post on this topic coming soon. But the long story short - we wanted exactly the attributes mentioned in the previous paragraph. And we did not succeed in our quest for such a tool. So we helped ourselves and developed it. After several years of testing performance of various <a href="http://www.redhat.com/products/jbossenterprisemiddleware/">Java Middleware products</a> we realized that our tool is really advanced and production ready. So let's open source it!
</p>
<p>
Whoa, young boys! We wanted to provide you something <a href="http://img2.netcarshow.com/Porsche-911_Carrera_4S_Coupe_2013_1024x768_wallpaper_01.jpg">really cool</a>. And we only had just the tool that did its job right, had all the attributes, but was only for <a href="http://www.carsbase.com/photo/BMW-Isetta_mp2_pic_43756.jpg">brave knight riders</a>. Over the nights, while our families were sleeping, we did a major rework of each of the part. Actually, this is still undergoing, but we are almost there.
</p>
<p>
Our main goal for this summer is to release the version 1.0. This includes a complete web site (all downloads working), complete documentation, the source code we can be proud of, high test coverage (all interface contract conditions must be tested for example), and a couple of success stories on this blog.
</p>
<p>
We would like to encourage you to give us some feedback on what you are missing in your favorite performance testing tool, to even become a member of our <a href="http://www.perfcake.org/community/">community</a>, or to send us <a href="https://github.com/PerfCake/PerfCake/issues?state=open">a bug report</a>. And we would be happy to help you use PerfCake in your environment, for you to be able to provide us a success story.
</p>
<p>
Feel free to try to reach us by any of the means mentioned on <a href="http://www.perfcake.org/community/">the community page</a>.
</p>Martin Večeřahttp://www.blogger.com/profile/01257832382088091783noreply@blogger.com0