Comparisons - Microsoft software and Iron Man!

I'm trying to understand Microsoft Operations Manager - it's a beast. It's something that is powerful, but is complex and at this point very mind numbing.

I'm starting with this and someone walked over to my desk and asked me "Whats going on?" I couldn't resist not comparing this with the beast in Iron Man.

The "real" Iron Man has a very nice finesse robotic suit - where as the beast - is over engineered and can't do everything that the "real Iron Mans" robotic suit can do not to mention the beast is ugly and has a huge footprint!

I say this - because when we are developing tools / software -we try to make it as flexible as possible. However, when you try to give someone a tool with every lever possible, you need to find a way to make sure it has some value right out of the box. Otherwise - all your left is with frustration and possibly loss of users, not to mention productivity!

Executive Salries - Can we please STOP talking about them!

It seems that the media can't let go of the salaries of some of the CEO's or management in companies. I for one am sick and tired of hearing about it.

You really have 2 options -
1. Find a person who can do a better job at a lower salary.
2. Become a board member of the company and do something about it.

I don't get the point why the media or people have to complain about someone elses salary. If someone is not doing their job fire them. If they are or protected you as an investor from significant losses, then they deserve their salary. If you think you can do a better job - go for it, convince the board of directors.

What I don't get is, why the hell are you talking about it day in and day out, when there are significant other topics to be discussed about! Some executives don't deserve their salary, but when the company is not doing anything about it - why are you - perhaps a non-investor so worried about someone elses pay!

Republicans killing the bailout

I have only one question - weren't most of these guys fired during this previous election.

Didn't they see what happened when a big company like Lehman went down. You really really dont cut off another hand when one of them is bleeding.

Update: Now they're going back to the TARP funds. Didn't the Democrats ask the White House to do that in the first place?

The Republicans can have a new mantra - "Republicans - Our Way or the Highway" or "Republicans - Ask Us How to Lower Your Net Worth"

Perl vs Java!

I had to write a simple program to send an email when a log file didn't have the right data. This log file is always apppended to - and the last line is what I needed.

That meant reading the file backwards - or perhaps from the end. I wanted to find something really simple in Java to do it. As much as I looked (FileInputStream, FileReader) etc, it all involved copious amounts of work. Since, I didn't know the length of each line, I couldn't easily go to the end of the file, and set an offset!

Here's one of the examples I found: readingfilebackwards for java.

Here's how I did it in perl:

open FILE "$file";
@lines = reverse ;
foreach $line(@lines)
{
do something
}

Simplicity simplicity simplicity!

Human Communication changes....

When the Mumbai blasts happened, one thing was new - the social networks. Twitter! The number of messages on #mumbai during the blasts, almost a blow by blow account. There are stories where people inside the hotel used twitter to find out if it was really true, and blasts were occuring. Twitter for communication!! Next generation of quicker communication! Move to SMS's and Twitter! Smart phones are changing the way we communicate with each other.

I'm excited to think of what the next generation of communication is going to be - while wondering - are we as humans going to lose the actual social network of sitting at a table and making a deal?

Mumbai Blasts

This was a little disconcerting on all fronts. A number of questions raced through my head.

1. How could after a parliment attack in Dec 2001, India be so woefully unprepared with the placement of the commandos?
1. Why should it take 9 hrs for commandos to get there?
1. Why on earth should there be only one unit based in one place for a country as large and populous as India?
and many more (and yes, they're all 1 because they're all screaming questions).

That said, after the initial 60 hours of gruelling, just by 10 people, we now have a situation where we're not doing anything. The reason the ISI or whoever keeps coming after us, is because theres a ridiculously poor response to an attack. Right now - theres a woeful lack of leadership in India. I heard there were protests in India with the slogans " A country of Lions lead by a a leadership of donkeys" (I paraphrase) - but the quote is such an accurate depiction of most of the leaders in the country today.

I don't know what to say at this point. I feel so helpless (for the lack of a better word) right now. I really wish I was in India. I wish I could be putting together some sort of team - a leadership team of sorts, to start a grass roots movement. May be not now, but in 5 years time, we could be in a better position to demand from our leaders, or put better leaders in place, or perhaps be leaders ourselves?

All that though - starts from the individual. The more I think about it, the more I believe that each individual that starts thinking about the betterment of their surroundings, starts talking to their friends and neighbours and starts forming groups and networks is what will be required to get a better country. It seems so simple - but when you put a diverse group of people - who's first thought on their minds are - how am I going to earn my lunch today - it gets immensely difficult.

The middle class per se - is already caught up in the material wealth gaining - so the middle class turns a blind eye, and when nothing else - bribes its way to what they need. We need to stop doing that. We need to ask for better leaders - and we need to start making sure that our kids can become better people!

Hardware at a Software problem?

I'm trying to get people to change their thinking. Should we really be throwing more hardware at inefficient code? Not all code is inefficient, but some of it is! What is the cost tradeoff? People say, it's ok to throw money at Hardware because it's cheaper than getting a human to fix the code and everybody is over worked and underpaid!

I've always thought that you fix problems at the software level, so that your software runs absolutely efficiently and performs better, but is a new way of thinking? Hardware is cheap (especially in the INTEL /AMD world as compared to the Sun world), developers are expensive? I understand that at one point that it will get significantly more expensive to change the hardware, but till then?

What would you do?

Faster Disks for Dbs?

I was having a conversation with my colleagues on Db performance, when it veered into Solid State Drives for TempDb. Assuming that since TempDb is very high writes and less reads, the faster the disks can write, the better it is, and the more you have the I/Os split up, the faster it will be.

That got me thinking - for places that manage a number of databases or even 1 with high volume, wouldn't it make significant difference if we drop in a solid state drive for Temp? This got me further thinking - if all Temp is expendable, and not really stored, why should it be a drive at all.

Why not use RAM Disk instead? Power outage - doesn't matter - since the data is all Temp anyways!

But what about taking it up a notch. (If it's not done already - I'm claiming rights to it :-) ) . Why not storage providers like EMC or others offer Tray's of just RAM for Temp storage? DBA's could potentially create the TempDb / Tablespace on the RAM Disks. When DBAs are managing large number of instances (Oracle) they could just as well point to this RAM disk in this tray.

Wouldn't that significantly improve performance? Instead of always looking at increasing I/Os on disks - especially for Temp, wouldn't this be a better way to go?

I'm writing this before any research on this topic, but I couldn't stop myself! Have any of you done any experiments with this? Do you know of / have any articles on this topic? If so - can you send me a link :)

India - Stark Contrasts!

I recently came back from my vacation in India. I love India. It's where I was born, where I studied and was raised for a better part of my life. Now that I stay in the US, and only go back for vacations and family visits, I can talk about how it has developed and perhaps not....

What I found in most places I traveled to was ...The roads need to get better. People need to be more civic minded. People need to realize that the Government cannot do everything! Ofcourse - you can also see that the Government employees are only interested in making sure that their wallets are well padded and more money flows under the table than in public. Thats for another discussion.

The reason I write this is is - I've visited a couple of Software Development shops there. They're awesome. Some of the larger ones have canteens to buy food at (not vending machines, but actual food) and a very nice atmosphere. I was surprised at the number of flat screens. I also visited a number of malls ( you cannot believe the number of these complexes that have sprouted up all over the city - even in almost villages!). You can get some of the finest designer items from Gucci to Versace to anything you name it you get it. Whats even more surprising is, that most of these shops have buyers - and most of them are middle class working men and women!

Here's where the stark contrast comes in. When you enter any of these malls - it feels like you've been transported into a different country. The creature comforts, the designer brands and what not. Exit that mall and you're back in the midst of bad roads and sometimes very polluted air. I think this tells more on the government of the city, state & country more than anything else. The people in India are very creative. They make the best of what they have. Yet - the Government does not reciprocate, their employees - all they do is pad their pockets with bribes, and spend that money in these malls, but no services are provided to the public.

Until that mentality changes, until the people demand more, get educated (they try, but a huge uneducated population is better for the politicians - give them a beer and ask them for their vote) and DEMAND more - I don't think the city is going to change.

They changed to an open market, because the country as a whole was pushed to the brink with it's debt and policies. I hope the country is not pushed again to the brink due to the lack of infrastructure. This is probably the best time for India to reinvest in the infrastructure - not just by private industry, but by the government too....

I guess I could have summed this whole post into one word - Infrastructure!

I wish I did have pictures though. It would definitely show the stark contrasts.

Another Open letter - call to arms for the Rescue package

Another open letter to Obama...something I wish he would say or something along these lines...

Dear American People: Today the Congress didn't pass a very important bill and I understand many of you called your Congressman / woman to oppose the bill because it's a bail out for Wall St & their bad investments. However unfortunate it is, what affects us affects them and vice versa. Today, the big banks and investment houses on Wall street are crumbling, because they are unable to get loans. What this means to us, the ordinary consumer, is that when we want to buy a car there wont be money available or when you want to get a loan to grow your small business, buy a tractor to farm your land or buy seed, the interest rate you will pay will be beyond the normal, and perhaps wont allow you to get a profit. The young people who are starting a family and need a loan to buy a house ...wont have the ability to get a mortgage. This is why this bill is important. While it looks ridiculous that we the tax payer, have to bear the burden, it is because the stakes are so high, that the government has to intervene. The Government must be the place where an ordinary American can go to get help, but it should also be the place where businesses come as a last resort.

Tonight, we must take this step to help these businesses, while keeping our core principles of oversight and protection for the tax payer we must make sure that the American financial system is better and healthier, and this is why this package is that important.
I urge you to call your representative in the house and senate and tell them that you will not let America fail and they should not let America fail.
You voted them to be your leader and representative in the government. Tell them to lead and represent you.

My letter to Obama wrt bailout.....

Hi Everyone,

I have seldom been participating vociferously in the current
election cycle (though people sitting with me at work may disagree ;-)
). However, with the current climate and Mcain's latest stunt, I
really wanted to put out something -- so that you may comment (I'm
also sending this as a suggestion to Senator Obama). Below is the
text of my email I sent to the campaign (I have no idea if they'll
even see it, but it sure trumps anything they're talking about so far
:-) )

Best Regards,

Murali

Sub: Alternative to Bailout
Dear Senator Obama,

Your response to McCain's political stunt is amazing, especially
within the context of the current economic crisis. The problem right
now is that there is an endless cycle of homeowners and communities a
lot losing value in their property, as well as losing their homes
because of foreclosure.

To stem the problem, you could / should propose something on the
following lines:

1: The markets have liquidity problems. So lets solve that by
injecting liquidity instead of buying the bad loans. (You could inject
50-100 billion dollars to give markets instant liquidity instead of
buying bad loans giving the Fed the authority
to buy GOOD commercial paper instead of bad assets (if they already don't have it) -- that will inject
liquidity instantly. The current problem is that banks aren't lending
because of their balance sheets, the Fed can surely lend and even make
a profit!)
2: Invest 200 billion dollars into easing the burden on home owners
by modifying bankruptcy laws (let judges modify the mortgage contracts) and converting bad mortgages into 30 yr fixed ones.
Those who can't show their income lose their home automatically to
avoid foreclosure. Only help those who have been cheated, those who
have gone into the homes with full knowledge need to be held
responsible for their actions.
3. Propose that people who can afford their homes but go into
foreclosure will be brought in front of a judge - this will zap the
republican mantra of "What about all those people who made the right
choice of not getting homes that they can't afford" (This and my
previous point)
4. Propose that people who do go into foreclosure MUST NOT destroy
the home property or will face legal criminal action (this will
prevent foreclosed homes in depreciating in value even further and can
be sold off quicker)

These investments coupled with a major push to resolve the underlying
crisis (by media, marketing and just American resolve), will I'm sure
get us out of the crisis much faster than anything GW or Mcain can
propose.

Sincerely,



PS: While I generally try to keep my comments just to tech, this is a much bigger issue which will touch everyone in almost any industry. So, I'm posting it :-). My faithful reader(s), please comment.

SPE - Software Performance Engineering

In my previous life as a performance engineer (I still am, but thats for later), my boss used to harp about the 7 steps of SPE. He would out of the blue, jump out and ask us - what is the SPE methodology and name the steps. Most of us would just stare out, ocassionally coming up with one or two of them (seldom in the right order).

It turned out I was reminded of it yesterday and was suggested I blog it. So - what are these famed 7 steps?

1. Assess performance risk
2. Identify critical use cases
3. Select key performance scenarios
4 Establish performance objectives
5. Construct performance models
6 Determine software resource requirements
7. Determine system resource requirements

Most of those are self explanatory. Funny thing is, in all the days I was there, we used to always try to get to that methodology and try to implement it. Ironically, we were doing it anyways, almost anybody who does anything with PerfEng has to end up doing it, but perhaps not that clearly defined.

The only one which most people may not do is Construct Performance Models -- I would replace that for existing systems with -- Construct a baseline performance model. For new systems - you can hypothesise away, but till you actually run a test, you're never going to get anything.

So - a tip of the imaginary hat to my old boss. And the bigger thing which I'm sure he'll be happy about is, that I do follow it to this day. It's insane to think that anybody who is in the PerfEng world wouldn't be doing any or all of it. To define it and to split it out into its own parts, is IMHO more academic than anything else. It's good to document those things, but in the long run, if you get stuck in the process, you don't have time to implement.

IIS 6 & Dynamic Compression!

Ok - this really got me! In my previous post, I spoke about the cool factor with regard to enabling compression. What got me was that our application dynamically generated Crystal Reports. Once I enabled compression, this feature broke entirely. It was a disaster. We had to roll back compression.

Looking at the code and everything else, I got nowhere! Finally, I decided to turn off compression for the those particular directories. Ohh...what a nightmare. As you can probably tell from my previous posts, I've always worked in the Apache-Tomcat/ Linux environment, and not so much in the IIS / Windows environment. Well....I got a crash course in understanding the Oh very flawed documentation on Microsoft's website (where they have disable instead of enable and vice versa for command examples - very confusing for a beginner let me tell you).

I tried running the adsutil.vbs script as suggested in the Microsoft docs directly

adsutil.vbs set /LM/W3SVC//root/directory1/directory2/DoDynamicCompression FALSE

This didn't do much. The compression was still taking place. Eventually though, it turned out that I had to make a seperate metabase entry for every directory that I didn't want to compress, since IIS had to get to it and after struggling with Metabase, I did the following.

For every directory I wanted to disable Dynamic Compressoin, I created a entry in the metabase.xml file.

<IIsWebDirectory Location ="/LM/W3SVC//root/directory1/directory2"
DoDynamicCompression="FALSE" >


You could use adsutil.vbs also to create the webdirectory and then disable dynamic compression
Adsutil.vbs create "/LM/W3SVC//root/directory1/directory2" "IISWebDirectory"
Adsutil.vbs set /LM/W3SVC//root/directory1/directory2/DoDynamicCompression FALSE

Ofcourse, if the directory requires any SSL or any such conditions, it may be lost. So, I checked the previous entries for the directories, and just copied it over instead of using adsutil.

Once I did that, it eventually stopped dynamic compression and things started running fine again. My Google searches led me to one author who actually said that there needs to be such an entry, and that was the "aaha" moment. It would have been nice for Microsoft to mention in their voluminous notes about this property, that IISWebdirectory needs to be created. In fall fairness, the Metabase property documentation did say "IISWebdirectory" but to a novice like me, I expected that if the "root" itself was defined as an IISWebDirectory, everything else would get picked up....


Well...that was my learning experience. Hopefully, the others who stumble upon this will have a less harrying experience then I have had!

Network Latency.....

When you have a global application, your servers cannot all be located in the same geolocation of all your users. Some users, will have to hop skip and jump to get to your app. While in most cases, this is ok - here's something to keep in mind about.

Huge Pages: By this I mean, pages which can grow over even 50 - 100K. Anytime you start entering that range (could be because of search results, or something else), you have to remember, network latency will start coming into play. Why? If the user is going to have to download a dynamic page or a large size, the time it takes to download will generally piss the user off. Thats why we have smaller images, better compressed images and what not (jpeg, png...)

But what about the regular run of the mill pages? The Asp, JSP pages? What about those pages when they get into the 500K range or in our latest case 3.9Mb!!!!

For a long long time now (~ 2004 or may be even earlier), Webservers have had the ability to zip up the data and send it across, and have the browsers unzip it. (Before Http 1.1, there used to be "deflate" and then now most of us use gzip). To get a webserver to do that, you need to let the Webserver know, you can accept zipped content and it's ok for the Webserver to send you such content.

Accept-Encoding: gzip, deflate

Thats what needs to be sent to the server to allow you to do that. Well..for IIS 6.0, this is not turned on by default - so you can imagine what happened, when we moved some pages to .net in our application and the "ViewState" variable grew incredibly large...we ended up with a 4Mb page which was being transmitted over the network across the atlantic to a user on a dsl connection.....haha! Guess what the user felt?! :)

The other thing to remember is, you have both "Static" (js, css, html) and dynamic content ( asp, jsp etc). When you do have static content, its obvious that you want a cached zip file to be sent out and after the first time, if you've coded it right, the browser shouldn't re-request the same file( Hint: Set the content-expiration header).

For Dynamic content - you're not going to get saved files, but think about it. If you have enough headroom on your server, and you are expecting large files, you could set this also (Make sure you have a baseline, so that if it gets too hard on the server, you can turn it off). You may also be able to do this at directory level to zip only certain directories.

If you really don't want your global users to feel pain - this is a setting you may want to turn on. IIS 7.0 I'm told comes with this turned on by default for static content.

According to some websites (and in my feeble attempt to find out about Apache) - I'm told that it is not turned on by default.

This would make a huge difference to your global users.

Again - I found this by using the Coradiant tool - to tell us that the damn page was 4Mb and the user was half way across the atlantic, and was on a dsl connection.

I deduced that we weren't using gzip and confirmed by using the YSlow plugin for Firefox.

Good luck! - and if my faithful reader(s) want to tell me about their experiences with this it would be nice! (Ofcourse, we may be behind the curve here and not have enabled it, but I'm curious on how many others have actually enabled it :-) )

Microsoft's LiveSearch cashback program

Ooohhh...is this the Google killer? Are they going to grab market share from Google? Apparently some of the analysts think so. Hahaha is all I can say. I won't be surprised if people search on Google for cash back and then go to the Microsoft's cashback site to take advantage of the offer. Thats still secondary. Here's the kicker....

I picked the very first thing I saw on Live Search - a Canon PowerShot A470 - gray and with Cash back, the lowest offer $107.31.

I did a Google Product search for the same thing - Cheapest Price - $97.99 from Amazon!

Here's the thing, you can offer people all the candy they want, but once the candy runs out, they're not going to stay for a second longer. It's like the politicians and their promises, it only lasts till election day.

So, for Microsoft to "disrupt" the model, I sure hope they come up with a better search engine, not better candy. Otherwise, all they're going to do is spend their money and get nothing for it in return.

Ohh - and by the way - from what I see on Microsoft's site, when the vendor charges you more, you get a bigger cash back (ofcourse, it's not enough to beat the lower price by any margin).

I think I'm going to stick to Google. I'm waiting for Google to announce it's next quarter results. It would be really nice to see Google announcing better returns and more searches, and Microsoft announcing ....well....!

Tools to Evaluate Web Design

I've spoken about how much I like coradiant, especially with the way they can capture all of the Web users session data etc.

On a different note, while thats good to evaluate the response time of pages etc, what about how much users actually like the webpage, or what they're doing on a webpage. I came across this posting from sixrevisions : 7 Incredibly Useful Tools for Evaluating a Web Design. Sixrevisions is going to be come a daily stop for me now!

I already use YSlow and it's pretty neat, especially when you want to track down whats taking the most time, or how best to optimize your webpage, but there are a few more tools that caught my eye.

Also, incase you miss it, there is a note about
Crazy Egg from one of the commenters on sixrevisons :

ncdeveloper

April 22nd, 2008

Crazy Egg is NOT for a production website by any means. I use it for the EPA websites and while it looks nice and flashy, it is not accurate at all after a few weeks. The counts just seem to go loopy after a few weeks. Support is just as weak. Every time I called, it sounded as if I woke the guy up or pulled him away from a tv show. Obviously a small company that was unprepared for moderate to high volume. If you have clients that want a real and accurate heatmap, look elswhere.


Please note, that that comment is there only because I'vnt tried it yet, but its also a fairly significant comment. I'm sure the reader(s) of this blog will understand that I'm in no way for or against using any of the tools on that site. I've tried only YSlow and I highly recommend it though :)

MySQL applications

Well, I've been using MySQL for the past few months now, and I guess, this list is again a good list to keep handy i.e. Handy Applications for a mysql database.

http://sixrevisions.com/tools/applications_mysql_databases/

Some really cool Web Developer tools

Ok....this got "Dugg" and is on my list. I've tried some of the tools provided on these pages, but some of the tools (FireBug & Web Developer for Firefox) are just amazing.

Here's a list of the Top 20 Useful tools for Web Development.
http://sixrevisions.com/tools/20_web_development_tools/

And if you do use IE (ughh)
http://sixrevisions.com/tools/internet_explorer_extensions_addons_web_developers/

Apparently - one of my colleagues who used the Microsoft Web Developer toolbar long time ago had tremendous trouble with it. He doesn't know how it is now.

Anyways - now there's more to look at.

SQL Statements - Always evaluate on the right side of the expression

While trying to tune a SQL Stored procedure, the DBA came across this statement....

select X from where id * -1 = id

That would mean that the optimizer doesn't know what index to use and that simple query took almost 1/2 second with 1000's of reads.

All he did was switch that around
select X from where id = id * -1

It executed in 0 seconds.

The entire stored procedure shed more than 1/2 the time it took to return data!

His simple statement: Always evaluate on the Right Hand Side!

Truesight TCP Timeout

So, this past week when looking at the TrueSight data from Coradiant, something jumped out at me. Though the data was being returned on the screen (from the Web app), truesight was reporting it as a Timeout (Server time out). This was odd.

I put in a call to the friendly Coradiant tech support. In 2 mins, the support tech got to the bottom of it. It turns out that TrueSight's TCP timeout was set to 30 seconds. Well, with our Ajax call, it just didn't work. Getting that time out set to 300 seconds (yes, horrible performance -- I know, we are working on that too) took care of it.

This information on how to make this change is not public, and must be done only by a Truesight tech. So, while its not something that we can change, it's something that the new Truesight image will atleast allow us to change!