Long time ago, i tied to discuss on measuring XACML PDP performance. As i mentioned earlier, it was little bit tricky task to define a proper test suit for XACML performance testing. But however, I have done some performance testing with WSO2 Identity Server, which is an open source XACML engine based on Balana. Using this blog post i would like to share my experiences on performance testing and the test resources that i used.
Pre – Request
JAVA JDK installation (jdk1.6.0_24 or latest)
WSO2 Identity Server 4.5.0 as XACML 3.0 PDP
Apache Jmeter 2.8 as testing tool as PEP
Getting Start
Following are the challenges that i came across.
1. Generating large number of different XACML policies. No of policies and complexity of them are the main metric that would effect for PDP performance. Therefore we need to create around more than 1000 XACML policies.I have created a simple policy generation tool for this purpose. Policy generation tool can be found here. For this tool, we need to feed a one base policy. By using this base policy, tool would create different XACML policies. More details can be found in README file in the SVN location.
2. Generating different XACML requests. PDP uses caching technology to improve the performance. If we use same XACML request to feed PDP, we can not measure actual PDP performance. Therefore we must load the PDP with different XACML request. We can use Apache Jmeter and feed different XACML requests by using a CVS file.
Request generation tool, that i have written, can be found here. For this tool, we need to feed attributes values for subject, resource, action and environment categories. Requests would be created with all combinations of above values. More details can be found in README file in the SVN location
3. Uploading XACML policies to PDP policy store in Bulk. Once we create more than 1000 policies, we need to import them in to PDP policy store. WSO2 Identity Server’s default policy store is a registry (database). Also management console does not allow to upload multiple policies. Therefore we need to write some tool upload policies in bulk. Here is the tool that i wrote. We can do it by automating the WSO2 Identity Server’s web service API. i.e “EntitlementPolicyAdminService”. You can find more details on this service from here.
4. Writing Jmeter script. If you are new to Apache jmeter, it is not easy to start load testing in a second. Because you need to consider few facts..
- WSO2 Identity Server PDP is exposed via web service API called “EntitlementService”. You need to call this web service API using Jmeter.
- “EntitlementService” is a secured service. You need to authenticate to access this service. For authenticate, we can use Basic authentication or “AuthenticationAdmin” API
- Once authenticated to “EntitlementService. We need to use the Cookie and send that Cookie in subsequent request.
- How to feed Jmeter with multiple XACML requests that are in CVS file.
Please find Jmeter script from here.
Steps to Setup
Step1. Setup Identity Server
- Configure MySQL database as WSO2 Identity Server’s data store. You need to configure master-datasources.xml file which can be found at <IS_HOME>/repository/conf/datasources. MySQL JDBC driver can be put in to <IS_HOME>/repository/components/lib directory.
- Increase memory foot print. You can configure memory using wso2server script file which can be found at <IS_HOME>/bin directory.
Step2. Create XACML policies and XACML requests using tool
Step3. Upload XACML policies in to Identity Server.
Step4. Configure Jmeter script.
- Upload jmeter script file.
- Configure location for CVS file that contains the generated XACML requests
Performance Test
I have carried out some long running test to see how WSO2 Identity Server has responded.
Test Environment
- Hardware : Machine with Intel (R) Core (TM) i7-3520M CPU @ 2.90GHz, 8GB RAM , OS 64bit Ubuntu 13.04
- One WSO2 Identity Server instance is connected with MySQL Database which is running in same machine. And memory foot print with -Xms1024m -Xmx2048m -XX:MaxPermSize=1024m. All others are default configuration.
- Jmeter is running in same machine with default configuration
Result
- 1000 concurrent threads
- 1000 Policies
- 10000 Different requests
- Decision cache time out 300 Seconds
- Transport : SOAP over HTTPS
- 8 hours long running test with 48 Million requests
Jmeter result – Almost 1800 TPS
CPU usages using Jconsole – Around 50 %
Memory usage using Jconsole – Around 1GB