Ultimate weapon to kill java memory leaks
You must be logged in to download
Introduction
A lot of programmers think that memory management is an old task from the C programming times. They believe that no memory control is required after Garbage Collector arrived to ours lives. That is almost true; Java is a good language that takes cares of this uncomfortable detail. But… always? NO.
It is very common to watch big installations at big customers that need to be rebooted every night, only to avoid this problem. But this is only a grandma solution. Memory leaks are usually known after java.lang.OutOfMemoryError, but this is not usually the problem: the main problem is the performance impact that this delivers to the affected platform during long time before the OutOfMemoryError is got.
A typical solution is the postmortem analysis. Although it sounds as a reasonable approach, it has a problem; analysis is some one who dead before: you need to wait until your system crashes in production time, grab a big file of 3 or 4 GB and put this information in an even bigger sized server, and take your time … no cafeine drinks are recommended.
Lucierna distributes a nice and smooth tiny solution: The Antorcha Memory Plumber. It is such an elegant solution because Memory Plumber only looks where the Java Memory Leaks usually are. Antorcha Memory Plumber monitors Collections and Maps, looking which one is growing up with no sense, and its full stack path, and reports this to you.
The installation process takes only two steps:
-Add “-javaagent:aspectjweaver.jar -cp AntorchaMemoryPlumber.jar” parameter in your jvm or app server startup script.
-Run your app under load conditions for 30 minutes and read the results from the stdout.
Installation examples
StandAlone Demo App
The usage of Antorcha Memory Plumber for a standalone Java application is easy: just download the zip file, uncompress it; there you will find everything required to run it properly, including third party libraries as well as licenses and some examples.
In order to provide you a quick start kit, a leaking class, called “abr” is included as well. In the next snapshot you can review the standard output that you get when running the “abr” leaking class:
WebApp running under App Server
Installation for any Java AppServer is easy, and basically requires the same: add the -javaagent with the AspectJ weaver, and put the Antorcha Memory Plumber in the class path, you can either use the ‘-cp’ option, or add to the CLASSPATH env variable. Depending on your application server the steps can be slightly different.
For Tomcat APP server here you are the details:
1) Copy the aspectjweaver.jar and antorchaMemoryPlumber.jar to any folder Tomcat user can access, for instance /opt/Tomcat/antorcha
2) Modify the startup script /opt/Tomcat/bin/startup.sh, including:
PRGDIR=`dirname "$PRG"` export JAVA_OPTS="-javaagent:/opt/Tomcat/antorcha/aspectjweaver.jar"
EXECUTABLE=catalina.sh
Then be sure to add to the class path the file /opt/Tomcat/antorcha/antorchaMemoryPlumber.jar, you can do it modifying the setclasspath.sh script or just copy it to TOMCAT_HOME/lib folder (as a quick approach).
How does Antorcha Memory Plumber work?
Antorcha Memory Plumber works using Java byte code instrumentation, based on AspectJ, to modify all the methods calling to collections and mapping operations. Tracking any addition, and then sampling those collections or maps that are constantly increasing. After some stages it verifies the object leaks, and provides you not only the leaking connection or the last method where the collection/map was increased, but also the full stack trace, so you can even easily identify leaks in Third Party Libraries, HttpSessionLeaks and so on.
As you can imagine, in order to allow Antorcha Memory Plumber to work, you will need to stress test your tool during the time that it is working (aprox. 27 minutes). The Full version of Antorcha Memory Plumber includes auto tune capabilities to setup proper heuristic thresholds like hunting time, etc; but with this “Light” version you should be able to find any leaking class always when you implement a Collection/Map interface (the most usual cause of memory leaks) and you achieve at least a total amount of 500 elements in that collection/map while Antorcha Memory Plumber is up.
Advanced configuration
Configuration parameters:
* amp.period.startup.delay: Minutes that the memory plumber will wait prior starting to search for leaks. If not set the default value is 2.
* amp.collectionelements.minimum: Minumum number of elements for that collection that will be needed to start to be considered as a leak. If not set the default value is 500
* amp.period.suspect: Time in minutes on where a collection will be treated as suspect, after that time, the collection is removed from the suspects if that collection decreases. If not set the default value is 10
* amp.iterations.numberof: Number of iterations on that the memory plumber will seek for any potential leak. If not set the default value is 3
* amp.iterations.period: Wait time in seconds per iteration. If not set the default value is 5
Usage example:
Setting a delay of 4 minutes before starting to look for leaks and a suspect period of 20 minutes, leaving the rest options to their default values.
java ….. -Damp.period.startup.delay=4 -Damp.period.suspect=20 …..
Download
Download is easy: if you are not a registered user you will be asked to register which is a very straightforward process: just include your e-mail, name, surname, country and company: that’s it. Once you are registered you have to log-in and then you will be able to download Antorcha Memory Plumber.
In case you find it useful, interesting, if you have doubts or whatever, please post a comment, your questions and feedback are always welcomed.
[...] think it can be very useful for those experiencing java.lang.OutOfMemoryError, aka: a memory leak: Ultimate weapon to kill java memory leaks Hope [...]
[...] Ultimate weapon to kill java memory leaks (tags: memory analysis aspectj thirdparty javabytecode via:zite) [...]
The download is incomplete, probably missing a jar file. :
Caused by: java.lang.ClassNotFoundException: com.lucierna.util.metrics.OverheadMeterer
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
I realized I upload the wrong version. I will upload and let you know within 1 hour aprox.
Thanks
Now version 0.81 removed unused references and should work.
Let me know how it goes and thank you for the feedback
thx for the fast repsons. The plumper works now. I’ve used it on a webapp with tomcat 6.0.32.
****************** Antorcha Memory Plumber RESULTS ********************
Leaking class: java.util.HashMap has 2783 elements. Some of them where added at org.apache.catalina.loader.WebappClassLo
ader.findResourceInternal(WebappClassLoader.java:3114) type: class java.util.HashMap and holds aproximately: -1 bytes. F
ull StackTrace:
null
Leaking class: org.apache.catalina.loader.WebappClassLoader$1 has 582 elements. Some of them where added at org.apache.c
atalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3054) type: class org.apache.catalina.loade
r.WebappClassLoader$1 and holds aproximately: 14550 bytes. Full StackTrace:
null
Leaking class: java.util.ArrayList has 3276 elements. Some of them where added at com.google.gwt.user.server.rpc.impl.Se
rverSerializationStreamWriter.append(ServerSerializationStreamWriter.java:579) type: class java.util.ArrayList and holds
aproximately: 26208 bytes. Full StackTrace:
null
Leaking class: java.util.IdentityHashMap has 936 elements. Some of them where added at com.google.gwt.user.client.rpc.im
pl.AbstractSerializationStreamWriter.saveIndexForObject(AbstractSerializationStreamWriter.java:199) type: class java.uti
l.IdentityHashMap and holds aproximately: 75816 bytes. Full StackTrace:
null
Leaking class: java.util.ArrayList has 2429 elements. Some of them where added at com.google.gwt.user.server.rpc.impl.Se
rverSerializationStreamReader.prepareToRead(ServerSerializationStreamReader.java:406) type: class java.util.ArrayList an
d holds aproximately: 19432 bytes. Full StackTrace:
null
**************** Antorcha Memory Plumber DEACTIVATED ******************
Now i will have to find out what is causing this.
Hi,
Am also facing the same issues when i run the Anchtorcha Memory Plumber tool.
***** Antorcha Memory Plumber: StackTraceHunting Stage Stopped
****************** Antorcha Memory Plumber RESULTS ********************
Leaking class: java.util.HashMap has 3944 elements. Some of them where added at
org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2135)
type: class java.util.HashMap and holds aproximately: -1 bytes. Full StackTrace:
java.lang.Thread.run(Thread.java:619)
***** Antorcha Memory Plumber: StackTraceHunting Stage Stopped
****************** Antorcha Memory Plumber RESULTS ********************
Leaking class: java.util.HashMap has 3944 elements. Some of them where added at
org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2135)
type: class java.util.HashMap and holds aproximately: -1 bytes. Full StackTrace:
java.lang.Thread.run(Thread.java:619)
May i know the cause of this issue and how to resolve the issues.
Thanks,
The key here is what Abraham said… to be able to interpret that information. I have similar output and the point now is to figure out whether this numbre od elements in memory is common or not. I am checking that mines are more or less the same so, do we have a memory-leak problem
?
By the way, thank oyu Abraham for your contribution to our work.
Which version of GWT you are using?
The one related to Tomcat, could be leaks, but are related to resource. As you can see in the code Antorcha points to:
// Add the entry in the local resource repository
synchronized (resourceEntries) {
// Ensures that all the threads which may be in a race to load
// a particular class all end up with the same ResourceEntry
// instance
ResourceEntry entry2 = (ResourceEntry) resourceEntries.get(name);
if (entry2 == null) {
// THIS IS THE LINE L-AMP POINTS:
resourceEntries.put(name, entry);
} else {
entry = entry2;
}
}
return entry;
}
if you review the rest of the code you would see, this map is never removed. However I believe the number of resources that can be cached there can not cause a real leak in your application although it is…
If you provide the GWT version we can have a look at it…
Hi!
i have the folow error when i try to run tha antorcha
Error opening zip file or JAR manifest missing : aspectjweaver.jar
Error occurred during initialization of VM
agent library failed to init: instrument
what i missed?
i have a webapp in tomcat 5
thank you!
Hi! It seems aspectjweaver.jar can not be located by you jvm. Try adding the fullpath of the file and check tomcat user has full rights to read and execute the file .
If this does not work for you: send me the details: where aspectj is located, as well as the options you are using to start tomcat. I’ll be happy to help you.
Feedback is welcome.
How is a memory leak defined ragarding this tool? A Collection/Map that gets to more than 500 elements while the plumber is up?
This one, the light version 0.81, has all the parameters hardcoded. Your collection/map needs to reach 500 elements during the first stage: you can see the timings reading the logs. Then it iterates over the discard stage, that removes all candidates that during sampling it’s number of elements decreased.
Afterwards it enters into the last added cycle to get the last addition stack trace and finally displays only collections and maps that were not filtered and that keeps owning more than 500 elements.
Memory plumber full version dynamically autotunes this parameters by looking at GC parameters.
Due to the high number of downloads and people positive feedback, we will release a new version shortly (here as well) where this parameters can be set up through java options, the code will be included as well and also extra documentation regarding how to tune.
If you have any suggestion they will be more than welcome.
Thanks.
It’s just that the accepeted definition of a memory leak reads as “an object that is stored in memory but is no longer accesible from running code”, so I fail to see how the number of elements of a collection is indicative of whether you have a memory leak or not, when the important point is that the fact that says that you have a ML is when you have an object that is no longer reachable from your code but it cannot be garbage collected.
I understand that growing collections can be “suspicious” but unless they are unreachable from your code and unable to be GCed, there is no memory leak.
I also fail to see how time has anything to do with it. In typical memory problems in web applications, ML occur when the context is restarted and that is independent of how long the application has been running.
Don’t get me wrong, I’m not attacking the tool but I’m just trying to understand which kind of problems it might help detecting and I’m just a bit confused because I think Memory Leak means something quite different in my book
.
Cheers!
Hi, thanks for your time and comments, I don’t get you wrong: all comments are welcome, specially when they can help clarify topics that maybe not clear, or why, not wrong.
First of all, let me introduce what we think a memory leak, according to wikipedia: “A memory leak, in computer science (or leakage, in this context), occurs when a computer program consumes memory but is unable to release it back to the operating system. In object-oriented programming, a memory leak happens when an object is stored in memory but cannot be accessed by the running code.[1] A memory leak has symptoms similar to a number of other problems (see below) and generally can only be diagnosed by a programmer with access to the program source code; however, many people refer to any unwanted increase in memory usage as a memory leak, though this is not strictly accurate from a technical perspective.”
As you can imagine, we consider memory leak as an unwanted increase in memory usage. In java particular case, as memory management is almost done by the JVM itself it is almost impossible to get a memory leak in java HEAP space (unless you are dealing during some time with a finalizer thread overload conditions), in fact there is no way you can not access an object from your code because that is the exact candidate the GC frees for you. So only the C Heap space could suffer a memory leak according to the “exact” definition. In oder words, unless JVM bug, or you are using native leaky code (like native DB drivers for instance) or because the very weird problem related to finalizer thread, there are not “exact memory leaks” in java HEAP according to the definition.
It is common to consider leaks in java as those patterns where you start adding elements to collections/maps that constantly increase. Of course it is really tricky to split those collections that does not increase indefinitely from those that constantly increase driving to a java.lang.OutOfMemoryError. For instance: imagine an online shop where all the products are stored for caching purposes in memory of course it is not a leak, although it could cause an OutOfMemory. Now imagine the same shop, that puts in memory, in a static collection for instance, all completed purchased, after some days, that collection will be using the effective memory and you will get and OutOfMemory, this is what we call a memory leak.
According to this example AMP light will show both collections, in one case it will be right, and in the other one (the first one) although a suspect it won’t be a leak.
By the way, we have included the source code, and added some parameters so you can make AMPLight more flexible, but as any other tool: it requires afterwards some human interpretation to get the best from it.
Regards.
Well, I don’t agree that you cannot get memory leaks in Java in the heap space. In fact you can and they are very common in what one could call “managed environments” where your program does not control the whole JVM and specially in places where you are dealing with Threads or Classloaders that your application does not own.
In fact, the very same JDK produces memory leak issues, for example if you create a custom log level, because the Log class registers automatically a reference to your log level class that you cannot control or remove, or in certain cases when you load some faulty JDBC drivers that self register in the list of available drivers but never remove themselves… etc.
Web applications is a whole different worl where memory leaks in HEAP are real and very common, usually due to buggy popular libraries (jakarta common are awful examples of that) and classloader and threading issues (bad usage of ThreadLocal mechanisms). Those are real memory leaks and are usually difficult to debug/work around, to the point that Tomcat 7 has added specific mechanisms to detect them and warn the user.
But what you are talking about are no leaks, there is no leaking because the application is still in control of those objects and the tool cannot decide if what you are seeing is normal or not, hence why you’ll get false positives in many cases, as the message from evdzwan above shows, where the supposed “leaks” are web app classloaders, identity arrays etc.
In any case, I think that “memory leak detection tool” is misleading and you’ll have plenty of people confused, specially if they come from the web development world. IMHO, it would be better to focus on what the tool exactly does.
Cheers!
According to your initial definition there is no way to have a memory leak in Java Heap Space. If the object is referenced, so it is accessible from some part of the code: either yours or from the managing environment but in any way accessed, it is not leaking. Antorcha does look in all classes no matter if they come from your app code or from your managing environment (as you have seen in the example you mention from other user) and in turns it points any potential leak.
I agree with you in one point: this tool, as well as any other tool to detect memory leak, can only provide a list of potential leaks, then an analysis needs to be performed to determine not only if there are leaks or not but also the severity to decide if you need to fix them or not, as I told you in the previous reply. However in order to be a little bit more precise: the difference between a leaking solution and unlimited growing collection is a matter of time and the usage of that collection/map.
For instance, regarding the ClassLoaders, there are several good app servers using hybrid technologies to free memory and dehydrate metada associated to JSPs and web apps, that are re-hydrated to memory when required, just to avoid an unlimited growing collection. In oder words, the example you took from edvzwan (org.apache.catalina.loader.WebappClassLoader.findResourceInterna) I can assure you can get a tomcat app server, just loading a huge amount of dummy resources collapsed with an OutOfMemoryError, not because of the bundles but because of the metadata the webappClassLoader is storing exactly at the point the tool showed. Of course it is not probable such a scenario so this is the type of potential leaks will almost never cause you any pain. But, regarding the same example what can you tell about the GWT potential leaks?
Anyway just to assure anyone reading this is not miss-confused by our approach or yours I will suggest here a nice reference, in fact the one Google returns when you look for “java memory leak” and press “I am feeling lucky”:
http://www.ibm.com/developerworks/library/j-leaks/
where the example it is based on, is one Antorcha Memory Plumber would have discovered easily.
Again, thanks for your time and please do not get me wrong as well, just adding more info about this.
plop!
You can get a memory leak in heap space in Java because when it says “the program” it means your application, not the whole JVM. The objects “could be” dereferenced and garbage collected by the JVM or your managed environment but they don’t know they have to as they don’t have the required logic to make that decission.
And I’m not claiming you cannot get an OOM by holding lots of objects in Collections such as a Classloader list. But calling that a memory leak in incorrect. An OOM simply means you needed more memory than what you had; it does not mean you have a leak. It might, but it could be simply a case that you misjudged how much memory you needed for your app.
Regarding the GWT potential leak, it looks like an index table to prevent duplicated when objects are serialised/deserialised through the RPC mechanism. So, no leak unless the GWT engine does not clean it appropriately when it should and even if it keeps growing it simply means that you are sending many objects and you need more memory to prevent duplicates or you need to change your RPC mechanism. But again, unless GWT is buggy, it’s no leak. It’s just memory consumption.
And if you follow your “nice” reference that Google returns, you will see that they indeed talk about a Memory Leak in the HEAP in Java and to define it, they don’t count collection members, they check objects that should no longer be there and still exist in HEAP because they are still referenced from places that prevent them to be GCed. If Antorcha Memory Plumber does that: great! that’s detecting MLs!
But anyway, I simply wanted to say even though I find it useful, if it simply uses element counting and size, claiming that this is the “ultimate weapon to kill java memory leaks” is misleading and it could backfire, But I have no hidden interests in the matter and no desire to discuss through the web, so good luck with it.
[...] antes de obtener un OutOfMemoryException. Esta disponible para su download gratuito en http://ctoblog.lucierna.com/ultimate-weapon-lucierna-kill-memory-leaks/ [...]
I am also having the same problem as laslorma:
Error opening zip file or JAR manifest missing : aspectjweaver.jar
I tried putting the file in every place I could think of, including bin\ and lib\ of tomcat and of the jdk. I tried using the full path with both forward and back slashes, and setting the file permission to Everyone:Full Control. This is using tomcat 6, java 1.6.0_26, and Windows 7.
I realize this is a java or tomcat problem, not Antorcha’s problem, but I haven’t been able to find a solution anywhere else. Any other ideas about how to fix? Did it ever work for laslorma?
THX
Windows is sometimes tricky to work with classpath and file separator. Here you are a demo sample for Windows:
set JAVA_OPTS=%JAVA_OPTS%:”-javaagent:C:\Software\apache-tomcat-6.0.33\antorcha\lib\aspectjweaver.jar”
Of course, class path need to be modified to include the AntorchaMemoryPluber.jar as well. Please let me know if this works for you or not, so we can help you to make it work.
Regards.
I figured it out finally – the aspectjweaver.jar and AntorchaMemoryPluber.jar files were encrypted. I don’t quite understand how they got that way, but once I decrypted them, everything worked fine.
THX
May I use this product commercially if I do not redistribute it or any derivative works to parties outside of the company that I registered as?
no error is displayed while running nor any output is displayed on screen.
I tried to run it with small executable jar with,
java -jar -javaagent:aspectjweaver.jar -cp AntorchaMemoryPlumber.jar someGenerator.jar
what am i missing
Ok. There are two reasons to get this behavior:
1) Your app (someGenerator.jar) does not use any collection/list, so the Memory Plumber is not started. This is the most probable thing.
2) Your app uses Collection/map/list or whatever, but it redirects standard output internally to some log file, so Memory Plumber does not show the results in the expected screen but yes in the proper file.
Does this make sense to you ? In case no, could I get the someGenerator.jar file to test internally?
Regards.
Hi..
I put the antorchaMemoryPlumber.jar in the tomcat lib, and my application war has the aspectweaver-1.6.7.jar, but i don’t see any antorchaMemoryPlumber logs although my application does create lots of collection obje cts in cache…not sure if i am missing something.
Hi Rahul, please see the next reply. It should help you.
Regards.
I just started the server and hit a URL which creates lots of collections internally but cannot see any antorchaMemoryPlumber logs.
Does antorchaMemoryPlumber shows it after 27 minutes?
Is there any way to see if the antorchaMemoryPlumber is active?
Sorry Rahul for the late reply … I forgot to do it before.
Basically if you are monitoring an application that create collections you should get an initial message telling you Antorcha Memory Plumber is active. If you are not getting this message, then it is not active at all.
Could you provide me more details about your configuration? Server type, start command, and paths where you deployed everything?
Thank and regards.
Hi Friends.
I use Glassfish v3. but I’m not sure the file and the lines I should modify!!…to run AntorchaMemoryPlumber….
Regards.
Hi David,
Here you are some extract on how to run antorcha on glassfish. You should follow the same procedure:
1.Copy the antorchaMemoryPlumber.jar library to the following path:
/{GLASSFISH_HOME}/glassfish/domains/domain1/lib/
.2. Access the administration console. To do this, firstly execute the application server:
———
cd {GLASSFISH_HOME}/bin/
./asadmin start-domain
Waiting for DAS to start ………
Started domain: domain1
Domain location: /opt/glassfishv3/glassfish/domains/domain1
Log file: /opt/glassfishv3/glassfish/domains/domain1/logs/server.log
Admin port for the domain: 4848
Command start-domain executed successfully.
——–
And then access the adminstration console: https:// GLASSFISHIP:ADMINPORT
3. Go to JVM Setting –> JVM General Settings, deselect the option Debug Enabled and erase the default parameters of the debug options. Then save and restart.
4. Go to JVM Setting –> JVM Options and enter the javaagent and antorcha home. Then save and restart.
“-javaagent:/ {GLASSFISH_HOME}/glassfish/domains/domain1/lib/aspectjweaver.jar –noverify”
5. In case you experience some issues related to log complains like:
Could not load Logmanager “com.sun.enterprise.server.logging.ServerLogManager”
java.lang.ClassNotFoundException: com.sun.enterprise.server.logging.ServerLogManager
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
Then add the options the next options also in the step 4:
-Dorg.aspectj.tracing.factory=default -Dorg.aspectj.tracing.file=/tmp/aspectj.log -Dorg.aspectj.tracing.enabled=false
That’s it, let me us know how it goes.
Regards.
Hi,
How do I run it for Java 1.4.
Thx
Use it with java 1.4 requires replacing the default classLoader with one that passes the byte code to aspectj in order to weave it which is it complicated if you are not used to do it. In case of Antorcha Memory Plumber Full we support it but I am afraid that part of the product is not open at all. In other words: Antorcha Memory Plumber Light only works for java 1.5+.
How i can installed on WebLogic Server 10.3.2.0 ?
Modify your startup.sh script, and add the next option:
export JAVA_OPTIONS=”$JAVA_OPTIONS -Xbootclasspath/p:C:\bea\weblogic92\samples\domains\wl_server\antorchaMP\antorchaMemoryPlumber.jar;C:\bea\weblogic92\samples\domains\wl_server\antorchaMP\aspectjweaver.jar -javaagent:C:\bea\weblogic92\samples\domains\wl_server\antorchaMP\aspectjweaver.jar -noverify”
Of course bare in mind the path convention is different if using a Unix like OS.
Let me know how it goes.
Hi
I am trying to use for Jboss 7.1 with linux OS.
I modified the set-up files as below…
Step:1 startup.sh
JBOSS_CLASSPATH=”$JBOSS_CLASSPATH:/home/jboss7.1/antorchMemoryPlumber/antorchaMemoryPlumber.jar”
Step2: startup.conf file
JAVA_OPTS=$JAVA_OPTS -javaagent:/home/jboss7.1/antorchMemoryPlumber/aspectjweaver.jar
When i tried attempt to re-start the server, getting below error, could u pls help me on resolving this….
—————————
WARNING: Failed to load the specified log manager class org.jboss.logmanager.LogManager
Exception in thread “main” java.lang.ExceptionInInitializerError
at org.jboss.as.server.Main.main(Main.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.modules.Module.run(Module.java:260)
at org.jboss.modules.Main.main(Main.java:291)
———————–
Thanks in advance
After several tests we got what’s going on with jboss 7. First there is a know conflict between aspects and some log implementations. To prevent it you have to add the next options:
JAVA_OPTS=”$JAVA_OPTS -Dorg.aspectj.tracing.enabled=true -Dorg.aspectj.tracing.factory=default -Dorg.aspectj.tracing.file=/opt/jboss7/antorcha/log/aspectj.log”
The next problem you may face is that with OSGi class loading it may be difficult for aspectjweaver to find the memoryplumber.jar. To avoid it just add the jar to the Xbootclasspath, something like this:
JAVA_OPTS=”$JAVA_OPTS -Xbootclasspath/a:/opt/jboss7/antorcha/memoryPlumberWhatever.jar”
Please let us know if it works for you!
Thank and regards.
thx abraham for repsonse…
Yes i implement the same, this time, jboss server got up, but no sign of log posted in neither /log/aspectj.log nor server.log
I even stress test (using silkperformer) the application under test, but still no sign of logs even after 27 mins…
To confirm now i have below settings got added only in standalone.conf of JbossAS7 for linux OS
#——————————————————————————————————-
JAVA_OPTS=$JAVA_OPTS -javaagent:/opt/jboss7/antorcha/aspectjweaver.jar
JAVA_OPTS=”$JAVA_OPTS -Dorg.aspectj.tracing.enabled=true -Dorg.aspectj.tracing.factory=default -Dorg.aspectj.tracing.file=/opt/jboss7/antorcha/log/aspectj.log”
JAVA_OPTS=”$JAVA_OPTS -Xbootclasspath/a:/opt/jboss7/antorcha/memoryPlumberWhatever.jar”
——————————————————————————————————–
And i created a log directory /opt/jboss7/antorcha/log/ with all permissions, for logs posting as suggested.
your timely help in this regard is highly appreciated.
Note: using jconsole, its very apparent that to see signs of memory leak, to figure out the exact cause, fully dependent on this tool.
Thanks and Regards,
Prathap
Just to be sure:
You did not put this, JAVA_OPTS=”$JAVA_OPTS -Xbootclasspath/a:/opt/jboss7/antorcha/memoryPlumberWhatever.jar”, right? or unless the jar: /memoryPlumberWhatever.jar exists? I meant you should point there to the memoryPlumber.jar file. I answer through iPhone with no access to the my laptop so I could not provide you the specific detail.
Sorry for the inconvenience…
Hi Prathap, finally we discovered why this is happening. jBoss 7 is real pain with the new modularization and estrange class loading hierarchy. The reason why Memory plumber is not telling anything is because the aop.xml inside the memory plumber is not being exposed to the other class loaders, so there is no instrumentation apart from the main module.
Here you are the generic way we have created in the AspectJ wiki to make LTWi work in jBoss 7:
——
If you have tried to activate LTW under jBoss 7 you will have found unless three problems that prevents it to work.
You get an exception of type and afterwards the server suddenly stops:
Exception in thread “main” java.lang.ExceptionInInitializerError
at org.jboss.as.server.Main.main(Main.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.modules.Module.run(Module.java:260)
at org.jboss.modules.Main.main(Main.java:291)
Caused by: java.lang.IllegalStateException: The LogManager was not properly installed (you must set the “java.util.logging.manager” system property to “org.jboss.logmanager.LogManager”)
at org.jboss.logmanager.Logger.getLogger(Logger.java:60)
at org.jboss.logmanager.log4j.BridgeRepositorySelector.(BridgeRepositorySelector.java:42)
Even if you have been able to make the server startup using some jBoss configuration options as suggested in the next link your aspects are not beeing injected where they should.
:The solution
The IllegalStateException is thrown by jBoss7 because there is a bug that prevents any class to the java.util.logging. You can see the acknowledged bug here. There, a partial solution to avoid this problem is suggested, regarding changing the way the loadmanager is loaded, pushing it to the BootClasspath and adding several configurations. However we suggest to remove AspectJ tracing facilities and preventing it to use through configuration to avoid it. You can achieve this by adding the next options: -Dorg.aspectj.tracing.enabled=false -Dorg.aspectj.tracing.factory=default
Due to new jBoss classloader and modularization architecture, classes stored in your javaagent are not visible to the remaining modules, so your aspects will not be found and you will get different types of errors. To let your aspects been found by all your code you have to add the aspectjweaver, and aspects.jar files to the booclasspath and add the next option to jboss startup: -Djboss.modules.system.pkgs=org.aspectj,com.yourcompany.aspects.pacakage that makes every class under those packages to be shared across all the modules in the jBoss system.
Tighting all together
Here you are an step by step guide:
Modify the jar contanining your aop.xml. Move that aop.xml somewhere inside your package structure, for instance com/company/myaspects/aop.xml and create the jar again as aspects.jar.
Modify jboss standalone.cnf file adding the next options to the end of it: JAVA_OPTS=”$JAVA_OPTS -Djboss.modules.system.pkgs=com.company.myaspects,org.aspectj -Dorg.aspectj.weaver.loadtime.configuration=com/company/myaspects/aop.xml -Xbootclasspath/a:/tmp/aspects.jar:/tmp/aspectjweaver.jar -javaagent:/tmp/aspectjweaver.jar -Dantorcha.install.dir=/tmp/antorcha -Dorg.aspectj.tracing.enabled=false -Dorg.aspectj.tracing.factory=default”
—–
If you find any trouble, please let me know.
Hi! I have been tryng to use this Memory plumber with the following command:
java -Dorg.aspectj.tracing.enabled=true -javaagent:/home/viana/Util/antorchaMP-light-v0.82/aspectjweaver.jar -cp /home/viana/Util/antorchaMP-light-v0.82/AntorchaMemoryPlumber.jar -jar OSSINRTProcessor.jar
My command-line stand-alone java application seems to run just fine, as I get the expected output on the stdout, however, I am not getting any indication that Memory Plumber has actually started….
Hi Gizmo,
How many time was your application running since start until it was finished?
getting error : [WebappClassLoader@1de6be9] warning parse definitions failed — (Exception) Unknown element while parsing element: aspectwerkz
Unknown element while parsing element: aspectwerkz
java.lang.Exception: Unknown element while parsing element: aspectwerkz
at org.aspectj.weaver.loadtime.definition.SimpleAOPParser.startElement(SimpleAOPParser.java:154)
at org.aspectj.weaver.loadtime.definition.SimpleAOPParser.traverse(SimpleAOPParser.java:190)
at org.aspectj.weaver.loadtime.definition.SimpleAOPParser.parse(SimpleAOPParser.java:68) ….
any idea whats the problemo ???
Hi!
Are you using aspectwerkz? It seems the application you are trying to debug uses some sort of aspectj inside that owns a file called aop.xml, inside some META-INF folder inside a jar, that includes a tag called aspectwerkz and is not a valid aop.xml.
If this is the case, I will work it out a workaround for you, but needs to confirm before.
Hi Abraham
I try deployed in JBoss-4.0.5.GA , but not sure whether Antorcha has started or not. I put in my jboss start script
JAVA_OPTS=”$JAVA_OPTS -Djava.net.preferIPv4Stack=true -javaagent:/opt/BV1TO1/JBoss/server/bv_framework0%2
Hi Abraham
I try deployed in JBoss-4.0.5.GA , but not sure whether Antorcha has started or not. I put in my jboss start script
JAVA_OPTS=”$JAVA_OPTS -Djava.net.preferIPv4Stack=true -javaagent:/opt/BV1TO1/JBoss/server/bv_framework0/lib/aspectjweaver.jar”
JBOSS_CLASSPATH=”/opt/u01/app/oracle/pro
Hi Ateh,
I think you should add the memory plumber jar to the classpath…. Does it make sense to you?
Hi Abraham
Thanks for reply
Yes, I did that too.( Actually I mention that on the comment above including the CLASSPATH and JAVA_OPTS parameter, but somehow all my comment lost when I press the Submit Comment button, it took more than 1 minute until lost connection, I tried several time).
beside that is there jar file I missing out?
thanks
Hi Ateh,
Sorry if I do not understand your question, but just to clarify:
You have added aspectjweaver.jar with the “-javaagent” option that is right.
But you should add also the file antorchaMemoryPlumber.jar to the classpath, either by adding a CLASSPATH entry or by including it in the the lib folder of your jboss. Something like /opt/jboss/server/default/lib/antorchaMemoryPlumber.jar.
Have you done this? In case no, please give it a try. In case yes, I will require some logs: server.log and stdout.log
Thanks and regards.
Hi All,
I need a help here . I was trying out the tool below is what I got java -javaagent:aspectjweaver.jar -cp ./antorchaMemoryPlumber.jar:. cs8000
Exception in thread “main” java.lang.VerifyError: Expecting a stackmap frame at branch target 60174 in method cs8000$ConnectionHandler.run()V at offset 7
at cs8000.(cs8000.java:12811)
at cs8000.main(cs8000.java:12774)
Hi!
Try adding the -noverify option to the JVM and let me know if it works.
Regards.
Hi,
I was succeeded in running Antorcha memory Plumber tool with Tomcat in windows machine.
But, Iam failed and unable to run Antorcha Memory Plumber tool in Ubuntu machine.
I tried the exact above steps mentioned in the topic “WebApp running under App Server” still its not running.
Even I executed the below command in root Console :
java -javaagent:aspectjweaver.jar -cp ./antorchaMemoryPlumber.jar
Can anyone please give me the proper steps how to install and configure with Tomcat in Ubuntu machine ???
Many Thanks,
Venkat
Hi Venkat,
I am not sure about this, but could you try to specify the full path of aspectjweaver.jar in your startup options, I mean something like:
java -javaagent:FULL_PATHTOASPECTJWEAVER/aspectjweaver.jar -cp FULLPATHTOANTORCHA/antorchaMemoryPlumber.jar
In negative case, could you provide catalina.out last lines while restarting to get a clue?