Large number of lwp_suspend calls.

Question

Hi,
i have a sample test program which moves files between directories. The code is pasted below.
The mv is taking more time on one of the servers compared to others.
A truss reveals an unusally high number of lwp_supend sys calls on the problematic server.
What could be the reason for this? How can we trouble shoot this.
public class Test {
      * @param args
     public static void main(String[] args) {
          if (args.length != 2) {
               System.out.println("Usage::  InputDir OutputDir ");
               return;
          String commandEx, commandSym;
          String inDir = args[0];
          File inDirFile = new File(inDir);
          String outDir = args[1];
          while (true) {
               File[] files = inDirFile.listFiles();
               if (files.length != 0) {
                    System.out.println("No of files = " + files.length);
                    Date startTime = Calendar.getInstance().getTime();
                    System.out.println("Time Before moving files is :" + startTime);
                    for (int j = 0; j < files.length; j++) {
                         commandEx = "/bin/mv " + files[j] + " " + outDir;
                         Runtime rt = Runtime.getRuntime();
                         Process result = null;
                         try {
                              result = rt.exec(commandEx);
                              result.waitFor();
                         } catch (Exception e) {
                              System.out
                                        .println("Encountered exception while trying to copy files");
                         if (0 != result.exitValue()) {
                              System.out.println("Failed to copy file");
                    Date endTime = Calendar.getInstance().getTime();
                    System.out.println("Time after moving all the files:"
                              + Calendar.getInstance().getTime());
                    long diff = endTime.getTime() - startTime.getTime();
                    System.out.println("Time to move: " + diff / 1000 + " secs");
               } else {
                    System.out.println("No files in the input directory");
                    return;
               return;
}

Answer 1

Quite frankly, I'm surprised this code isn't causing a noticeable headache on all your machines, if any of them. It's hard to imagine they are all the same kind of system, with comparable loads, but only one is struggling with this task.
How many CPUs on the system? If your load averages, as shown in the prstat output you gave, exceed the number of CPUs, the box in question is probably saturated.
I'd first be curious how many LWPs are attributable to the Java process itself. I'm finding it hard to take my eyes off this section:
                    for (int j = 0; j < files.length; j++) {
                         commandEx = "/bin/mv " + files[j] + " " + outDir;
                         Runtime rt = Runtime.getRuntime();
                         Process result = null;
The code draws one Runtime object for every mv(1). That can scale to fork-bomb status depending on the number of files you're moving. And since each separate process runs in its own subshell, it's going to fight all other concurrent processes for access to the target directory's lock, in order to place and record its file entry safely. The many lwp_suspend calls may be attributable to all but one of those processes being told to wait until the directory lock is freed by the current owner.
It would be good for all systems to economize on the action here. Why this approach would afflict one system but not others that are effectively the same is a puzzler.
I'd suggest looking at prstat -L, but with 16k+ LWPs it's going to output a small phonebook of entries.