Monday, 20 July 2009

Dave Matthews Band - Big Whiskey and the Groogrux King

My favourite group, Dave Matthews Band released their 7th studio album - "Big Whiskey and the Groogrux King". I feel this is their best album since 1998's "Before These Crowded Streets" with them going back to their jamming style of music rather than the manufactured pop sounds of recent albums.
The name is a tribute to founding member and saxophonist LeRoi Moore who died in August 2008 and his nickname was Groogrux. A few of the songs on the album have references to LeRoi and show that the band really miss their mate. The cover portrays Moore's head on a float in the Mardi Gras procession in New Orleans.





The album begins with a short and lovely saxphone solo "Grux" - recorded by LeRoi before his death.

This is the prelude to the first song "Shake Me Like a Monkey" which is one of the outstanding tracks. The lyrics exhort a babe so hot "that God and the devil alone could not have made you up. Two must have worked as one" to "Love Me Baby Shake Me Like a Monkey". and this gem "I like my coffee with toast and jelly. But I'd rather be licking from your back to your belly". Very peppy and a kick-ass blast this song is.

"Funny the Way It Is" is a poem showing the same events has opposite effects on different people. "Hear the laughter while the children play war" and "Funny the way it is, If you think about it, One kid walks ten miles to school, another's dropping out"
The words remind me of Alanis Morissette's "Ironic" and the tune is simple and catchy with subtle violin at the start and a full blooded violin interlude in the middle with strong rock guitar riffs. I think this song also shows the band misses LeRoi with this line "Somebody's heart is broken and it becomes your favorite song". I dont like the video too much though.


The third song "Lying in the Hands of God" is a slow ballad with a simple arrangement. Hummable with reflective lyrics. This song grows on you with it's slow guitar rounding off - and you can picture yourself looking into a fading sunset with a drink in hand.

Then there's another decibel upping blast "Why I Am" - starts off with a full-on drums and guitar rolls and is definitely again a tibute to LeRoi "Why I am still here dancing with the GrooGrux King, We’ll be drinking Big Whiskey while we dance and sing" and you do want to tap along with the dance. This is surely an instant hit.

"Dive In" is my 2nd favourite song on the album - the one I loved as soon I heard it. It starts on a slow note but Dave sounds hypnotical in the lilting tune singing an ode to all things Summer - Summer games, Summer breeze, Summer girls, Summer dresses - "Kick off your shoes, dive in the empty ocean". "Wake up sleepy head, I think the suns a little brighter today". It surely is guys - after listening to this - my day has brightened. Wake up and enjoy it while it lasts cause it wont last forever - is the message here.


"Spaceman" is an okay song compared to the others. If it were on one of the weaker earlier albums, Busted Stuff [2002] or Stand Up [2005], this might have been rated higher but for me, it just makes up the numbers on this album. the lyrics are about a guy's love for his girl, even when things dont work out.

"Squirm" is quite unusual and surprising. It starts off slow but punches you when Dave gets a bit menacing and the crescendo builds as he sings "Drumbeats louder (x 4)". He gets ferocious on the "Open up your head" chorus. Parts of it almost feels like a song by U2, I can imagine Bono cooing in falsetto between the climactic violin and drum rolls, and it blends in what sounds quite Middle Eastern wailing for the finale. There's much more meaning to these lyrics with it's referring to (lack of) Faith and the differences in Religions and the inevitable End. But I love the song for it's music rather than it's philosophy.

"Alligator Pie (Cockadile)" is my favourite song on the album. It's arranged chaos is sublime - listen closely for the dog barking just before the violin in the beginning and Dave's own howling bringing in the vocals. This song will get your feet tapping with the banjo driving the entire melody. On the face of it, this is a song for his daughter Stella asking "Daddy when you going to put me in a song" since many earlier songs have referenced her twin sister Grace. (Yup, even I dont get why didnt her put her in a song earlier) But Dave's also pushes in a critical reference to the Bush administration's response to the Hurrican Katrina devestation of New Orleans. The song proceeds into a lovely cacophony with Dave almost yelping through the words. My only complaint is it ended too soon at 3:57 mins. The title word "cockadile" is the child's pronunciation of "crocodile" but I did not hear the word used in the song. Must be an in-family joke.

"Seven" begins with a heavy guitar sound almost like Pearl Jam and then moves into another jamming event. Most of the song is in falsetto with plenty of naughty lyrics. I wonder if the title itself alludes to The Seven Minutes- the novel by Irving Wallace. But it's fun to hear this all the same - and you'll start humming it in no time.

"Time Bomb" begins so similar to "The Dreaming Tree" from "Before These Crowded Streets". Soft alto saxophone and ballady tune "I'm a ticking time bomb Waiting to blow my top" and - Woah it explodes into a powerful metal guitar as if Metallica guested on this after 2:40 mins. Dave's shouting out angrily and covered by a strong trumpet "I want to pick up the pieces, Hammer in the final nail, I wanna believe in Jesus". Yes this is a dark song about death and killing and the purpose of it all. This song echoes very strongly in your heart if you have experienced loss - either by death or a failed relationship.
There are some lyrics which possibly refer to the death pf Dave's older sister, Anne, who lived in South Africa, was murdered by her husband, who subsequently committed suicide, on or around January 27, 1994. "No one would believe it
He was such a normal guy
Shake their heads and wonder why?
"
I feel the subtext here is on the existence of God and whether we are controlled by a higher power "If Martians fell from the sky what would that do to God? ".

When you're recovering from that rant, the soft guitar chimes in like a refreshing glass of chilled water and thats the opening of "My Baby Blue". This is the best ballad on the album - a love song dedicated to the one true love who he seems to have lost "Holding tight to the legs of all your angels, Goodbye my love ". Talking about the blue eyes, I feel though this is about loss, it's not dedicated to LeRoi, and there are some forums suggesting this is about his sister Anne's death. In a way this seems like he is at peace with the loss that was experienced in "Time Bomb" and I personally feel this is a follow-on to Time Bomb. This is an 'easy listening' track for Pop fans.

"You and Me" is another simple and easy ballad. This is probably a dedication to his wife - "When the kids are old enough we're gonna teach them how to fly. You and me together, We can do anything baby". This is mostly in acoustic guitar and rounds off the album on a soft note.


Yes - This is a lovely collection and took some time coming.

Cheers to Dave, LeRoi, Stefan Lessard, Carter Beauford, Boyd Tinsley, Jeff Coffin and Tim Reynolds with a Big Whiskey in hand.

Monday, 13 July 2009

JVM Tuning from the Trenches

This article is a follow-up to http://jojovedder.blogspot.com/2009/06/slow-weblogic-part-6-jvm-heap-analysis.html. Please read that one first for the basics on JVM heap, parameters and flags.

Also remember these tips have worked for the server settings and issues faced below, but blindly using these on your server will not produce the same results. You have to measure and then tune after measuring.

Problem:
Platform running Weblogic 8.1 on Sun V880 servers. Total RAM of 32 Gb on the machine.
2 Gb assigned to the managed server JVM heap. JDK 1.4

Initial settings:
-XX:+AggressiveHeap -Xms2048m -Xmx2048m  -XX:SurvivorRatio=32 -XX:MaxPermSize=128m 


But still there are 20 Full GCs per hour in peak times, before the server crashes.


Analysis

1. It was decided to reduce the SurvivorRatio to 4 and restart with some more flags.

The size of ONE Survivor Space is calculated as

SurvivorSpace = NewSize / (SurvivorRatio + 2)

Keeping SurvivorRatio as 32 means the Survivor spaces are too small for promoting stuff from Eden. Hence we reduce this to 4 which allows for larger Survivor spaces.

2. As per Sun Bug ID: 6218833, setting AggressiveHeap set before Heapsize (Xmx and Xms) can confuse the JVM. Revert the order to have -Xms and -Xmx to come before -XX:+AggressiveHeap or not use it

3. The application has 180+ EJBs with pools of beans. Hence set the -Dsun.rmi.dgc.client.gcInterval=3600000 (1 hour) instead of the default 60000 (1 min). More on this here: http://docs.sun.com/source/817-2180-10/pt_chap5.html

4. The site is restarted once a week at 4:30AM. The patterns stays normal for 2 days – and then degrades into full GC.

5. The Old space is pretty much full – at every minor collection – the Old space must be cleared up for promotion from Young to Old to take place.

6. Permanent space is pretty much full – keeps loading classes and classes ( could that be a problem – the difference between the number of JSP’s per Release?)
Hence we increased the PermSpace from 128M to 256M

7. Ensure we are running the server JVM by using the -server flag

8. Use OptimizeIt or similar profiling tool to see the memory usage and find code bottlenecks.


The settings now were

-server -Xms2048m -Xmx2048m  -XX:MaxNewSize=512m -XX:NewSize=512m -XX:SurvivorRatio=4 -XX:MaxPermSize=256m -Xincgc -XX:+DisableExplicitGC -XX:+AggressiveHeap -XX:-OmitStackTraceInFastThrow



This reduced the Full GCs to one a day.

Error Logs

At the time of the server going out of memory prior to a crash, the logs are filled with repeated errors (up to 100 repetitions) of this sort

java.lang.NullPointerException
 <<no stack trace available>>


Adding the -XX:-OmitStackTraceInFastThrow flag resolves this problem, the root cause of the NPE it self has to be tracked down but we do not have any longer the issue of huge recursive exception strings.

We could now see the stack trace as

java.lang.NullPointerException
 at java.util.StringTokenizer.(StringTokenizer.java:117)
 at java.util.StringTokenizer.(StringTokenizer.java:133)
 at jsp_servlet._framework._security.__login._jspService(login.jsp:294)
 at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)
 at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1075)


This seems to be a Sun bug described here.

Tuesday, 7 July 2009

More on Weblogic Muxers

This is a follow-up to an earlier article which detailed the Weblogic Socket Muxer pattern in thread dumps.


Another ERROR pattern associated with Muxers is shown below


####<Jun 23, 2009 9:25:24 AM BST> <Error> <Socket> <d001> <managed1> <ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>>
<BEA-000403> <IOException occurred on socket: Socket[addr=/10.23.22.1,port=1950,localport=61002]
java.net.SocketException: Connection refused.
java.net.SocketException: Connection refused
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:672)
at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:648)
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:123)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)



As written in an earlier post, each incoming request on a TCP socket uses one File Descriptor.

We monitored the File Descriptors available to this Weblogic server using



Here the Weblogic pid = 8359

/usr/proc/bin/pfiles 8359 | grep rlimit


1024

ls /proc/8359/fd | wc –l


140



The first command shows that 1024 FDs are allocated to this process, and the second shows how many are being used in real-time.

A plot of the second value during high load times shows the FDs go up to 900+ in use and hence a resolution used in this case is to increase the number of FDs.

A recommended value is to increase it to 8192.