Tuesday, May 26, 2009

Step for migrating AIX to AIX 5.3


AIX  5.3 Migration Checklist and Procedure steps

 

 

 

 


Steps for Migrating a server to AIX 5.3 (from AIX 4.3.3, AIX 5.1, or AIX 5.2):

Hide details for Prior to the migration, check the level of microcode on the CDROM drive and upgrade it if necessary. (downlevePrior to the migration, check the level of microcode on the CDROM drive and upgrade it if necessary. (downlevel CDROM microc...
Prior to the migration, check the level of microcode on the CDROM drive and upgrade it if necessary. (downlevel CDROM microcode can cause a 0C45 hang). No boot required:

  1. Do "lscfg -vpl cd0"
    1. if the part number is 04N2964 or the FRU is 04N2967 or the FRU is 97H7796, this is a 40X cdrom (ROS Level and ID show the microcode level) (1_06 is current)
    2. if the part number is 97H7795 or the FRU is 97H7610 or the FRU is 97H7796, this is a 32X cdrom (ROS Level and ID show the microcode level) (1_03 is current)
  2. Make sure that there are no cds in the cdrom drive
  3. Mount /inst.images from codeman.
    1. If this system is in the blue zone, issue mount codeman:/inst.images /inst.images on the system
    2. If this system is not in the blue zone, issue ssh_mount /inst.images system_name /inst.images on codeman
  4. for the 40x cdrom "cp -p /inst.images/microcode/cdrom/40x_web/* /tmp"
  5. for the 32x cdrom "cp -p /inst.images/microcode/cdrom/32x_web/* /tmp"
  6. for both, you also need the genucode stuff. "cp -p /inst.images/microcode/genucode/web_microcode/* /tmp"
  7. unmount /inst.images
    1. If this system is in the blue zone, issue umount /inst.images on the system
    2. If this system is not in the blue zone, issue ssh_umount /inst.images system_name on codeman
  8. issue cd /tmp
  9. now run "/tmp/cpcat"
  10. Then run "/tmp/genucode -s cd0".
  11. Wait a couple of minutes for the internal processing on the cdrom to complete.
  12. issue rmdev -dl cd0
  13. issue cfgmgr
  14. the command lscfg -vpl cd0 should now show "ROS Level and ID" of 1_03 for 32x or 1_06 for 40x.
  15. you do not need to reboot to make this effective.



 

Migration:

  1. Run checkout and ensure that it is clean (see U8BA Midrange System Support)
  2. Do a mksysb or sysback backup to tape NOW.
  3. To capture the migration, do all of the system steps from here on using autocons and set up a "script" capture. This will allow us to review the entire migration, in case of problems. To do this, logon to any system that is set up to run the autocons command (codeman, any of the autocons machines, or your AIX machine, if it is set up). Then issue:
    1. script system_to_migrate.migrationlog
    2. autocons system_to_migrate
    • At any time, if the migration has failed and you want to see the log, enter "~." to exit autocons and then use cntrl-D to close the script file. You can now view the system_to_migrate.migrationlog
  1. Run /hone/support/pre_aix53
    • This does a bunch of stuff, including removing some stuff that can foul up the migration and saving some parameters for the post install. It also unmirrors rootvg (MUST be done). It will stop every time that it does anything major or runs into a problem, so you can't just start it and walk away.
    • Be sure to document each time that it stops. Some items will need to be fixed after the migration (such as putting wu-ftp back into /etc/inetd.conf)
    • unmirrorvg and reducevg can sometimes hang.
      • If you need to kill them, you will probably need to reboot before you can do any recovery. If rootvg contained hdisk0 and hdisk1 before the unmirrorvg/reducevg and lsvg -p rootvg shows something like:
        • hdisk0 active ...
        • hdisk1 ??? ...
      • then you will need to run:
        • redefinevg -d hdisk0 rootvg
        • synclvodm -v rootvg
      • Once you have fixed the problems with unmirrorvg/reducevg, rerun /hone/support/pre_aix53
  2. Reboot to clean up the unmirror and other items. Use shutdown -Fr.
    • Warning: Sometimes login via the CDE gui fails after this procedure. You may need to use line-mode.
  3. If using CD's:
    1. Be sure that you have the right level of CDs: AIX 5.3 - TL 5 - CD's should be LCD4-7463-06 (dated 7/2006)
    2. Place CD #1 into the machine
    3. Reboot using shutdown -Fr, but don't allow the machine to go through a normal boot-up sequence. Press F5 (or 5 from autocons) when you get the keyboard prompt.
  1. If using NIM:

Hide details for NIM InstructionsNIM Instructions

    • NIM Instructions
    1. On the migrating machine:

                                                         i.            run bootinfo -p to determine hardware platform type: chrp rspc rs6k (circle one) ONLY chrp is supported

                                                       ii.            run lscfg | grep proc to determine up (one processor) or mp (multiple processors): up mp (circle one)

                                                      iii.            run /hone/support/ipandsna.pl | more

        • Default Gateway (Default Route) = ________________
        • IP = _______________________
        • Mask = __________________
        • MAC Address (optional) = ________________________
        • For ethernet, media_speed = _________________________
        • Slot = ____________________
        • Ethernet or Token Ring: ________________
        • Note: If the machine is on both the 9.17.201 and 9.17.205 networks, you can configure NIM to use either one. However, you need to be consistent and use the IP name for the correct adapter and the right gateway (ex: hone1d for 9.17.205 with gateway of 9.17.205.1 OR hone1 for 9.17.201 with gateway of 9.17.201.1)
    1. On moondance:

                                 .            smitty nim_mkmac

a.       Enter the name of the machine to be migrated

b.      If this is a new network, you will be asked to choose eth or tok, as appropriate for the machine being migrated

c.       Set "Hardware Platform Type" to chrp

d.      Set "Kernel to use for Network Boot" to either mp or up

e.       NIM Network

          • If machine network is known to NIM (already configured), it will automatically be filled in with something like ent_NetworkX.
          • Otherwise, you will be creating a new network and will need to set the correct subnet mask and gateway
          • The "Network Adapter Hardware Address" is not required. If you want to set it, use the MAC address that you got earlier

                                                                                                         i.            smitty nim_bosinst

 .        choose the name of the machine to be migrated

a.       choose rte

b.      choose 5305_lpp_source

c.       choose 5305_spot

d.      Choose the "BOSINST_DATA to use during installation": aix53_migrate_bosinst

e.       set "ACCEPT new license agreements" to "yes"

f.        Change "Initiate reboot and installation now?" to "no"

g.       set "ACCEPT new license agreements" in the "installp Flags" section to "yes"

        • If you receive an error:
          • 0042-001 nim: processing error encountered on "master":
          • 0042-302 m_allocate: the state of "buckskin" prevents this operation
          • from succeeding. Use the "reset" operation to correct
          • its state then retry the intended operation.
          • Use smitty_nim_mac_op to issue the reset. Change "deallocate" to "yes" and "force" to "yes"
          • Then go back and do this step (smitty nim_mac_res)
    1. Reboot your migrating machine using shutdown -Fr, but don't allow the machine to go through a normal boot-up sequence. Press F1 (or 1 from autocons) when you get the keyboard prompt.
    2. Select Multiboot
    3. Select Boot Sequence
    4. Make the correct adapter the 1st device
    5. Back out to the main SMS menu
    6. Select Utilities
    7. Select RIPL (Remote IPL Progam Load)

                                 .            Set the IP information

        • Define the IP of the migrating machine (client)
        • Server is 9.17.205.67 (moondance)
        • Gateway is the correct gateway to reach the server
        • Set the correct network mask

                                                                                 i.            If ethernet, set the speed and duplex for the correct adapter in Configuration or Adapter Parameters

                                                       ii.            Ping Test

        • Select the correct adapter
    1. Exit all of the way out of SMS to allow the bootp to start
    2. Note: You sometimes will see a double NIM/Sysback boot. This is normal and has to do with firmware interaction with AIX 5.3.

                                 .            LOAD: Waiting 60 seconds for Spanning Tree
BOOTP R = 1 BOOTP S = 1
FILE: /tftpboot/lmtaix07
Load Addr=0x4000 Max Size=0x1ffc000
FINAL Packet Count = 16554   Final File Size = 8475417 bytes.

LOAD: Waiting 60 seconds for Spanning Tree
BOOTP R = 1 BOOTP S = 1
FILE: /tftpboot/lmtaix07
Load Addr=0x4000 Max Size=0xbfc000
FINAL Packet Count = 16554   Final File Size = 8475417 bytes.

  1.  
  2.  
  3.  
  4.  
  5.  
  6.  
  7.  
  8. We've encountered a number of problems with the migration. To help the support center, we will run all migrations in "debug" mode. Follow these instructions:
    1. First prompt will be to choose the console. Press 1 from an ascii console or F1 from a graphics, then enter.
    2. Second prompt will be to choose a language. Press 1 and enter for English.
    3. (if you chose "aix53_migrate_bosinst" in NIM, you can skip this step because debug mode is already on): You will now see a menu for the installation. Enter 911 and press enter now. This sets up debug mode.
    4. You will see the same installation menu again. Take the defaults.
    5. The next menu should say "migration". If it doesn't, stop and get help. Otherwise, take the defaults.
    6. The migration will run for a little while, perhaps 5 or 10 minutes. Don't walk away. There is another prompt coming up. Choose the option to continue with the migration.
    7. (for NIM, ignore this:) First CD will take an hour or 2. After that it will call for the CDs pretty quickly.
    8. When migration is complete, system will reboot itself. It may give a message about the Trusted Computing Base not being active. Ignore this. We don't want the Trusted Computing Base.
  1. If using CD's:
    1. If running on ascii console, first screen after reboot will ask for console type. For autocons, respond "vt100".
    2. After reboot, the system will ask you to accept the license. Always accept the license. Then quit all of the way out.
    3. Installation assistant will come up. You can go right to the bottom option that says "tasks completed, exit to login".
    4. Remove the remaining CD.
  1. Run /hone/support/post_aix53
    1. This does a bunch of stuff, including fixing some stuff that got fouled up in the migration.. It also mirrors rootvg if the pre_aix53 script unmirrored them. You can use topas to see the I/O is occurring during the mirroring (for a warm, fuzzy feeling). Note that this script will stop every time that it does anything major or runs into a problem, so you can't just start it and walk away.
    2. Be sure to document each time that it stops. Some items will need to be fixed by hand after the post migration script completes. If you are not sure, send an email to Richard with the items that you wrote down.
    3. lppchk problems - If the post processing shows lppchk problems, you should run lppchk -v -m3 2>&1 | more to get the full list of what is wrong. The post processing only gives you a short list. Then you will need to determine whether filesets need to be installed or removed.
    4. Note: you may get an error that devices.pci.14109f00 is at 5.3.0.0 instead of 5.3.0.10. This is a known problem and the fix is not yet available.
  1. Reboot using shutdown -Fr
  2. Check the clock on the system. (date command). Is it correct?
  3. Run checkout.
    1. Many things will have changed:

                                 .            Old>AIXLEVEL: 4.3.3.0 New>AIXLEVEL: 5.3.0.0

                                                         i.            no options

                                                       ii.            nfso options

                                                      iii.            paging may have moved (if you originally were mirrored an booted from hdisk1)

                                                     iv.            PS: root rpc.statd  (rpc.statd now runs under daemon instead of root)

                                                       v.            NETINFO - IP addresses should be the same, but AIX 5.2 reports on card positions differently.

                                                     vi.            SYS0 may have some changes

                                                    vii.            sendmail

                                                  viii.            inetd

    1. If you are missing other processes or disks or anything else, you should check this carefully.
    2. Update checkout's control file using the "-update" option. See U8BA Midrange System Support
  1. Run checkout -new
    1. AIX 5.3 has new processes, new filesystems, etc. Decide which ones to add to the checkout table.
  1. Run /hone/support/ckdump.pl. AIX 5.3 may need more dump space than AIX 4.3.3 did. You may need to increase dump space or /var.
  2. Update the dept database (BLDINET on notes) and the CM Integrator record for this machine to show that it is AIX 5.3 ML 5.
  3. Perform security scan:
    1. If MSS system, to get a scan on demand you have to engage the MSSD team directly, Brian Stough is the contact
    2. If not MSS system:

                                 .            bring up mozilla on new server: mozilla

        • If mozilla is not installed, use inst_mozilla to install it Database 'LURA Midrange System Support', View 'Procs\All By Doctype', Document 'inst_mozilla - Install the Mozilla Browser'
        • (you can use VNC to get a graphics session on the new server if you are working from a PC and have no Xwindows server)

                                                         i.            go to scanondemand URL https://scanondemand.secintel.ibm.com/

                                                       ii.            login and tell it to scan the machine you are on.

                                                      iii.            Ensure ALL vulnerabilities in the scan report are addressed! Once the scan is clean, copy the scan to the security archive directory as: /var/adm/sec_logs/archive/initial_scan

  1. Close the migration log file. Enter "~." to exit autocons and then use cntrl-D to close the script file. You can now view the system_to_migrate.migrationlog
  2. You are done. Be happy.

Hide details for Office Network Media SpeedOffice Network Media Speed

  1. During the migration to AIX 5.2 , some of the group have encountered problems with NFS (very slow response). I have contacted IBM support and it appears that we have a 'Media_Speed' negotiation problem between the machine network adapter and the Ethernet closet port switch. I have contacted the Boulder Campus folks and they have requested that we set

our office workstations to 'Auto_Negotiation' instead of '100_Full_Duplex'. This appears to correct the NFS response issue.

Please let me know if you, if you have any questions or problems and we can work with the campus folks to get your machine
communicating correctly.

Procedure for changing media speed:
When you issue the detach to your adapter you will lose your 'Default Gateway'. Record the gateway info from smitty tcpip 'Minimum Configuration & Startup' or from ' netstat -rn' (Your default gateway is listed in the second column to the right of 'default')

  • umount any NFS filesystems
  • ifconfig enx detach
  • ifconfig entx detach
  • chdev -l entx -a media_speed='Auto_Negotiation'
  • smitty tcpip, Select 'Minumum Configuration & Startup', Select enx and update 'Default Gateway
  • entstat -d entx Shows stats on ethernet adapter and Media speed selected and Media speed running (bottom of output). Also if you show any 'CRC Errors' or 'DMA Overrun' counts then this is a indication of a mismatch between the Ethernet adapter and the switch port.






Hide details for Things that can go wrong and how to fix them:Things that can go wrong and how to fix them:
Things that can go wrong and how to fix them:

  • openssh won't start (startsrc -s sshd gives a message that "sshd subsystem is not on file")
    • Try running /hone/support/ssh_aix53 2>&1 | more. If you get an error message from the openssl install that says something like:

Getting failed dependencies
/bin/sh
/bin/libc.a (shr.0)
/usr/bin/perl is not found

      • You need to run "updtvpkg" to fix the rpm packaging. Warning, when I ran this, it hung or looped. After 20 minutes, I had to find the last child process (use ps -ef | grep updtvpkg and then pidlist xxxx) and kill just that process (without -9). That allowed it to continue and finish.
  • Kerberos is downlevel (can cause openssh to fail). The "krb5*" filesets must be at 1.3 or above. 1.0 or 1.1 will not work on AIX 5.x.
  • Migration hangs at about 93% complete. Some parts of the migration can take quite a while. But, if the migration hangs at some point for 30 to 60 minutes, then something has gone wrong with the migration. Rebooting may fail at this point. (E105 is one symptom). Possible solution(s):
    • First try:
      • boot into maintenance, before mounting FS
      • .
      • #fsck -y /dev/hd4
      • #fsck -y /dev/hd3
      • #fsck -y /dev/hd2
      • #fsck -y /dev/hd1
      • #fsck -y /dev/hd9var
      • #exit
      •  
      • #bosboot -ad /dev/hdisk0
      •  
      • # sync; sync; sync; reboot
    • If that doesn't work, try:
      • boot into maintenance, before mounting FS
      • .
      • #fsck -y /dev/hd4
      • #fsck -y /dev/hd3
      • #fsck -y /dev/hd2
      • #fsck -y /dev/hd1
      • #fsck -y /dev/hd9var
      • #/usr/sbin/logform /dev/hd8; Destroy?YES
      • #exit
      •  
      • #df -k
      • nothing full nothing close to 90%
      •  
      • #lslv -m hd5 hdisk0
      •  
      • # lppchk -v, -c, -l -> all clear
      •  
      • #rmlv hd5
      • #chpv -c hdisk0
      • #bootinfo -B hdisk0 -> 1
      • #mklv -y hd5 -a e -t boot rootvg 2 hdisk0
      • #bosboot -ad /dev/hdisk0
      • .
      • #bootlist -m normal -o -> hdisk0
      • # sync; sync; sync; reboot

 

Posted by at