Benchmarking of Aida/Web 5.4 on Swazoo 2.0 with WAPT tool

Using WAPT Load Testing Tool

Goal: to compare Aida performance to Seaside and Ruby on Rails, as Jim Robertson is doing, see his blog here, here and here.

AidaCounterExample is made, which is as close as possible to Seaside counter example, which is a base for Jim's benchmarks

• Test: WAPT Capacity/Overral Performance test with default setup for 20 concurrent users, instead of Runtime which is increased to 5 min.
• Test scenario: open a counter example, increment it to 4, decrement it back to 0 (together 9 page requests) 

Test results and comparison

• local = WAPT running on the same machine as test target, on VMWare Windows XP
• LAN = WAPT running on Win XP notebook, connected via ethernet to 100M router/access point Lynksys and then to test machine
• tuned = VW memory tuned (ObjectMemory sizesAtStartup: #(10.0 10.0 5.0 10.0 5.0 5.0 5.0) as Jim Robertson did
• tuning for Aida didn't cause noticeable difference, so Aida tests are run on non-tuned image
• test machine: 3.2GHz P4, 2GB RAM, on Linux (OpenSuse 10.0)
• each test is run from complete reinstall on a fresh image, to ensure the same starting conditions (memory etc)

Notes from IRC chat with Jim Robertson (jarober)

18:03] jarober> 5 mins, like the ones I posted earlier
18:04] jarober> 11,289 total pages
18:04] jarober> avg sessions/sec: 37.7
18:05] jarober> avg pages/sec: 37.6
18:05] jarober> starting pages/sec: 39.1
18:05] jarober> ending pages/sec: 35.8

18:24] michaell: it uses redirection

19:51] strange is that i got 170 req/sec, but that is page /counter.html only
19:52] jarober> different hardware/lan
19:52] and for a single session
19:52] localhost only, 3.2GHz P4

19:57] jarober> 20 simulated users with WATP

20:46] jarober wow, Swazoo 2.0 is extremly faster to swazoo 1.0 on this counter example!
20:46] you are using swazoo 1.0, because you loaded it from parcels
20:48] 400 req/sec vs. 170 on 1.0

23:17] if i put 100 iteerations for 1 user, i get up to 270 req/s
23:17] if i put default 1 for 20 users, i got even less that you, about 20
23:18] jarober> I'm using the download of WATP, only allows 20 users
23:19] jarober> I just took the defaults on the Capacity test
23:21] jarober> I created a script that launches the Seaside Counter
23:21] jarober> Then increments to 4, then decrements to 0

23:22] inAida case you login in initialzation phase?
23:23] in PRofle Initial
23:23] jarober> I took that outof the script
23:23] yep, that's not good because it dont stay for test too
23:24] i sent you a new parcels and i'm sending you one more for counter xample
23:24] this one set security so you dont need login
23:24] jarober> ok

01:27] jarober, results on Aida so far: 37.462 pages, 7,76 sessions/sec, 125 pages/sec, 124 start, 124 end, 0 errors
01:28] 0 errors is important to note, be sure that others have 0 too, otherwise can be result higher, falsely of course

01:51] what is even more interesting, i'm just running 360 pages/sec :)
01:52] and CPU is still not saturated to the end

01:57] that's important, yess, but both me and jarober are testing on local machine right now
01:57] and we are far from tcp botlenecks yet IMO
01:59] no, I was using my LAN
01:59] running WATP on a Windows box, hitting apps on the Mac
01:59] running on a 100 mbit LAN
02:00] ok, that's can be different but still i think any of us is fast enough to make a difference
02:00] if cpu is not saturated and ethernet is not saturated, then clearly there's lag somewhere :)
02:00] well, this test is also to be undestand first
02:01] default settings don't saturate the server, at least mine no
02:02] jarober, i'm increasing Iterations, to 10 currently, and User think time is disabled
02:03] this give me 370 pages/sec, 0 errors