ARPACK/PARPACK Checkpointing Capabilities
-----------------------------------------

Hernan G. Arango
Institute of Marine and Coastal Sciences
Rutgers University
August 30, 2005

Checkpointing is a necessary capability for applications that use the
ARPACK library to solve very large (and usually expensive) eigenploblems.
There is not such capability in the released ARPACK (Version 2.4) library.
Although, in the contribution area there are a couple of routines modified
to do such task. The strategy was to add extra arguments to reverse
communications driver DSAUPD.  The restart is then triggered by setting the
IDO flag to -2.

This is not an optimal way to restart the ARPACK library because it is
not supported by all the drivers and does not ensure the same results.
I have not been able to get my applications to produce the exact results
in the same number of iterations when compared with an interrupted
computation.  The reason for this is that ARPACK has a lot of internal
parameters that are saved during the computations with the Fortran 
"save" command. This suggests that when an application is restarted from
a checkpointing file we are not solving exactly the same subspace problem.

An alternative checkpointing strategy is proposed that maintain the
integrity of the current ARPACK library.  There are not extra arguments
to IRAM. The uninterrupted capabilities of ARPACK is intact and the user
gets identical results. The new strategy is as follows:

1)  Move all variables declared with the "save" statement to common
    blocks in several include files:

    i_aupd.h   private variables used by _AUPD routines
    idaup2.h   private variables used by _AUP2 double precision routines
    isaup2.h   private variables used by _AUP2 single precision routines
    idaitr.h   private variables used by _AITR double precision routines
    isaitr.h   private variables used by _AITR single precision routines

    The declaration of these variables were removed from the ARPACK
    version 2.4 symmetric and non-symmetric routines and put in the above
    include files.  I am only interested in the symmetric and non-symmetric
    routines but similar strategy can be used in the other routines.

    A single include file is not possible because the same variable names
    are repeated with different meanings and passed as arguments to other
    routines. The renaming and changing argument names is not desired 
    because require a lot of testing and can break the current capability.

    For example, idaup2.h becomes

c
c     %---------------------------------------------------%
c     | Private variables used by  _AITR single precision |
c     | routines are saved in common blocks to facilitate |
c     | checkpointing.  All  these  variables  need to be |
c     | saved and recovered during checkpointing restart. |
c     %---------------------------------------------------%
c
      logical
     &           orth1, orth2, rstart, step3, step4
      common /lsaitr/
     &           orth1, orth2, rstart, step3, step4

      integer
     &           ierr, ipj, irj, ivj, iter, itry, j, msglvl
      common /isaitr/
     &           ierr, ipj, irj, ivj, iter, itry, j, msglvl

      double precision
     &           betaj, ovfl, rnorm1, safmin, smlnum, ulp, unfl, wnorm
      common /rsaitr/
     &           betaj, ovfl, rnorm1, safmin, smlnum, ulp, unfl, wnorm


    Notice that logical, integer and floating point variables are in
    separated in different common blocks.  This is very important in
    various computer architectures.

2) As in previous checkpointing attempts, the IDO flag with a value of
   -2 is used to trigger restart from a checkpointing file.

3) The only executable change done to the ARPACK routines it to add
   an extra condition to the initialization of several routines from

      if (ido .eq. 0) then

   to

      if ((ido .eq. 0) .or. (ido .eq. -2)) then


   Also, in the same IF block an extra conditional is added for the
   initialization of several variables during cold start and NOT
   restart. If restart, all these value are recovered from the
   checkpointing file.

4) The user now have access to all the internal parameters of ARPACK
   and fine tune its values to a wide variety of restart applications.

   Since there are repeated variable names in the above common blocks,
   the user need to address the common block in a compact way.  For
   example, in my ARPACK application module I added the following
   statements:

      integer  :: iaitr(8), iaup2(8), iaupd(20)
      logical  :: laitr(5), laup2(5)
      real(r8) :: raitr(8), raup2(2)
!
      common /i_aupd/ iaupd
#ifdef DOUBLE_PRECISION
      common /idaitr/ iaitr
      common /ldaitr/ laitr
      common /rdaitr/ raitr
      common /idaup2/ iaup2
      common /ldaup2/ laup2
      common /rdaup2/ raup2
#else
      common /isaitr/ iaitr
      common /lsaitr/ laitr
      common /rsaitr/ raitr
      common /isaup2/ iaup2
      common /lsaup2/ laup2
      common /rsaup2/ raup2
#endif

    This compact form facilitates the writing of the checkpointing file. 
    My checkpointing file is written in a NetCDF file for both serial and
    parallel applications.  The user also need to save all the scalars
    and arrays passed as argument to any of the _AUPD routines.

5)  The following routines were modified for checkpointing:

    symmetric:

    ARPACK/SRC/dsaupd.f            ARPACK/PARPACK/SRC/MPI/pdsaupd.f
    ARPACK/SRC/ssaupd.f            ARPACK/PARPACK/SRC/MPI/pssaupd.f

    ARPACK/SRC/dsaup2.f            ARPACK/PARPACK/SRC/MPI/pdsaup2.f
    ARPACK/SRC/ssaup2.f            ARPACK/PARPACK/SRC/MPI/pssaup2.f

    ARPACK/SRC/dsaitr.f            ARPACK/PARPACK/SRC/MPI/pdsaitr.f
    ARPACK/SRC/ssaitr.f            ARPACK/PARPACK/SRC/MPI/pssaitr.f

    non-symmetric:

    ARPACK/SRC/dnaupd.f            ARPACK/PARPACK/SRC/MPI/pdnaupd.f
    ARPACK/SRC/snaupd.f            ARPACK/PARPACK/SRC/MPI/psnaupd.f

    ARPACK/SRC/dnaup2.f            ARPACK/PARPACK/SRC/MPI/pdnaup2.f
    ARPACK/SRC/snaup2.f            ARPACK/PARPACK/SRC/MPI/psnaup2.f

    ARPACK/SRC/dnaitr.f            ARPACK/PARPACK/SRC/MPI/pdnaitr.f
    ARPACK/SRC/snaitr.f            ARPACK/PARPACK/SRC/MPI/psnaitr.f

    The modified files are included in the following tar file:

        restart.tar.gz
