AIX Solaris HPUX UNIX Linux system storage administration ksh/perl scripting

Yup, another PowerHA bug! cl_chfs: VG sharedvg is concurrent

admin — Sat, 19 May 2012 03:42:57 +0000

Problem:

When trying to extend a shared filesystem in Enhanced-Capable Concurrent mode VG in PowerHA, I always return the following message:

cl_chfs: VG sharedvg is concurrent

No matter I use smit cspoc -> filesystem

or command:

/usr/es/sbin/cluster/sbin/cl_chfs -cspoc -nnode1,node2 -a size=+100G -A no /usr/opt/db2/dat1

/usr/es/sbin/cluster/cspoc/cli_chfs -a size=+100G /usr/opt/db2/dat1

Solution:

The problem is due to a typo in line 139 of /usr/es/sbin/cluster/utilities/clresactive

if print -- "$lsvg_out" | grep -i -q "passive_only"

It should be

  if print -- "$lsvg_out" | grep -i -q "passive-only"

Please note it should be _ instead of – between passive and only.

Modify the /usr/es/sbin/cluster/utilities/clresactive on all the Power HA nodes, now extending sharded filesystem works!

Determining if TSAMP/RSCT rebooted a node/server

admin — Thu, 17 May 2012 03:13:10 +0000

Problem(Abstract)

This document will discuss the various ways to determine if RSCT has rebooted your node.

Symptom

Node reboots without operator issuing the reboot command

Resolving the problem

TSAMP cannot reboot your node, there is no functionality built into the core TSAMP application that allows for this functionality. However, RSCT (Reliable, Scalable Cluster Technology), the cluster provider that TSAMP ‘rides’ upon can and will reboot your node given a few different situations. This technote will not discuss why the node was rebooted, only some of the ways to determine if RSCT was the culprit who initiated the reboot.

Syslogs:

The easiest way to see that RSCT has rebooted a node is to check your syslogs for the following message:
Jan 11 10:15:38 node03 ConfigRM[1418]: (Recorded using libct_ffdc.a cv 2):::Error ID: :::Reference ID: :::Template ID: 0:::Details File: :::Location: RSCT,PeerDomain.C,1.99.25.16,18148 :::CONFIGRM_NOQUORUM_ER#012The operational quorum state of the active peer domain has changed to NO_QUORUM. #012This indicates that recovery of cluster resources can no longer occur and that #012the node may be rebooted or halted in order to ensure that critical resources #012are released so that they can be recovered by another sub-domain that may have #012operational quorum.

Jan 11 10:15:38 node03 ConfigRM[1418]: (Recorded using libct_ffdc.a cv 2):::Error ID: :::Reference ID: :::Template ID: 0:::Details File: :::Location: RSCT,PeerDomain.C,1.99.25.16,21028 :::CONFIGRM_REBOOTOS_ER#012The operating system is being rebooted to ensure that critical resources are #012stopped so that another sub-domain that has operational quorum may recover #012these resources without causing corruption or conflict.

The easiest way to do this is to grep “REBOOTOS” from your syslogs output file.

ConfigRM Trace File:
Your IBM.ConfigRM trace file can be found in the following directory and needs to be formatted to be read:
To format the file:
rpttr -odtic /var/ct/IW/log/mc/IBM.ConfigRM trace > /tmp/IBM.ConfigRM_Trace.out
** Note: Your specific file names for the trace file may differ from what is shown above **

01/11/2011 10:15:38 AM.886775 T(11844464) _CFD !!!!!!!!!!!!!!!!! PeerDomainRcp::haltOS Entered. !!!!!!!!!!!!!!!!!!!!!
/var/ct/IW/log/mc/IBM.ConfigRM/
01/11/2011 10:15:38 AM.886856 T(11844464) _CFD logerr: In file=/project/spreljan/build/rjans002a/src/rsct/rm/ConfigRM/PeerDomain.C (Version=1.99.25.16 Line=21028) :
CONFIGRM_REBOOTOS_ER
The operating system is being rebooted to ensure that critical resources are
stopped so that another sub-domain that has operational quorum may recover
these resources without causing corruption or conflict.

After running the above rpttr command to format the traces, grep the word “rebootos” from the output file.

Error Report (AIX only):
Create or view the error report with the following command:
errpt -a > /tmp/error_report.out

Search for the following message:
-----------------------------------------------------------------------
LABEL: KERNEL_PANIC
IDENTIFIER: 225E3B63

Date/Time: Fri Sep 9 17:35:14 2011
Sequence Number: 163592
Machine Id: 00C54C6E4C00
Node Id: ccdev32
Class: S
Type: TEMP
WPAR: Global
Resource Name: PANIC

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
ASSERT STRING

PANIC STRING
RSCT reboot caused by critical resource protection
-----------------------------------------------------------------------

On both AIX 5.3 and AIX 6.1 the “IDENTIFIER” is the same value so searching the error report for “225E3B63″ will locate and identify the RSCT reboot.

Event Viewer (Windows only):
Search the event viewer for “0xDEADDEAD” to find the RSCT initiated reboots.

GPFS filesystem outage with a kernel panic 225E3B63

admin — Thu, 17 May 2012 03:10:55 +0000

A kernel panic has occurred on a member host that is due to a IBM® General Parallel File System (GPFS™) trigger. The trigger repeats on a sporadic but recurring basis.

Symptoms

The output of the db2instance -list command includes a pending failback operation, as shown in the following example:

ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME -- ---- ----- --------- ------------ ----- ---------------- ------------ ------- 0 MEMBER WAITING_FOR_FAILBACK hostA hostB NO 0 1 hostB-ib0 1 MEMBER STARTED hostB hostB NO 0 0 hostB-ib0 2 MEMBER STARTED hostC hostC NO 0 0 hostC-ib0 128 CF PRIMARY hostD hostD NO - 0 hostD-ib0 129 CF PEER hostE hostE NO - 0 hostE-ib0

HOSTNAME STATE INSTANCE_STOPPED ALERT
——– —– —————- —–
hostA INACTIVE NO YES
hostB ACTIVE NO NO
hostC ACTIVE NO NO
hostD ACTIVE NO NO
hostE ACTIVE NO NO

In the previous example, hostA has a state of INACTIVE, and an ALERT field is marked as YES. This output of the db2instance -list command is seen when hostA is offline or rebooting. Since the home host for member 0, hostA is offline, member 0 has failed over to hostB. Member 0 is now waiting to failback to its home host, as indicated by the WAITING_FOR_FAILBACK state. After hostA is rebooted from the panic, member 1 will fail back to hostA.

Diagnosis

When you check the db2diag log file, you can find many log entries that indicate that a restart light operation has occurred, as shown in the following example:

2009-08-27-23.37.52.416270-240 I6733A457 LEVEL: Event PID : 1093874 TID : 1 KTID : 2461779 PROC : db2star2 INSTANCE: NODE : 000 HOSTNAME: hostB EDUID : 1 FUNCTION: DB2 UDB, base sys utilities, DB2StartMain, probe:3368 MESSAGE : Idle process taken over by member DATA #1 : Database Partition Number, PD_TYPE_NODE, 2 bytes 996 DATA #2 : Database Partition Number, PD_TYPE_NODE, 2 bytes 0

Another way to diagnose this type of problem is to check the system log. Run the OS command errpt -a to view the contents of the AIX® errpt system log. In the AIX errpt log, you might see log entries similar in the following example, which is for hostA:

LABEL: KERNEL_PANIC IDENTIFIER: 225E3B63

Date/Time: Mon May 26 08:02:03 EDT 2008
Sequence Number: 976
Machine Id: 0006DA8AD700
Node Id: hostA
Class: S
Type: TEMP
Resource Name: PANIC

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
ASSERT STRING
5.1: xmemout succeeded rc=d

PANIC STRING
kx.C:2024:0:0:04A53FA8::advObjP == ofP->advLkObjP

If you see a

KERNEL_PANIClog entry as shown in the previous example, the system reboot might be due to an operating system kernel panic that was triggered by a problem in the GPFS subsystem. A kernel panic and system reboot can be the result of excessive processor usage or heavy paging on the system when the GPFS daemons do not receive enough system resources to perform critical tasks. If you experience GPFS filesystem outages that are related to kernel panics, the underlying processor usage or paging issues must be resolved first. If you cannot resolve the underlying issues, run the db2support command for the database with the -s parameter to collect diagnostic information and contact IBM Technical Support.

DBI1122E after using db2iupdt command on existing instance

admin — Sun, 29 Apr 2012 21:37:34 +0000

Problem(Abstract)

db2iupdt on existing instance fails with DBI1122E on cleanly brought down instance.

Symptom

DBI1122E Instance db2inst1 cannot be updated or upgraded.

Cause

An attempt was made to either update or upgrade an instance. The error message is indicating an incorrect version. This is caused by the .instuse file being corrupted.

/db2inst1/sqllib/ctrl/.instuse

Diagnosing the problem

Run db2iupdt with debug option -d.
db2iupdt -d db2inst1Check if output contains db2version as 199 just before DBI1122E is received.
## exit function query_instuse_string
## INSTUSE_VALUE for INSTVER is 0
## db2version=199 <======
## db2proddir=UNKNOWN
## exit function chk_version
DBI1122E Instance db2inst1 cannot be updated or upgraded.

Check the content of /sqllib/ctrl/.instusecat /home/db2inst1/sqllib/ctrl/.instuse
If the file contains following, then .instuse file is corrupted

Resolving the problem

To resolve the issue, perform the following steps.

1. Login as instance owner

2. Manually edit the .instuse file to enter following fields:

BitWidth=64

RealBitWidth=64

InstanceType=1

InstallerID=0

INSTVER=97

V=9

R=7

M=0

F=1

I=0

S=3

SpecialBuild=

Once, step 4 is run, above will be corrected depending on the latest fixpack user installed.

3. Login as root
4. Run db2iupdt again.

Related information

db2iupdt
DBI1122E

Oracle installation compiler requirement

admin — Fri, 27 Apr 2012 02:34:55 +0000

Compiler Requirements for Oracle Database 11g Release 2(11.2) installation:

The following is the minimum compiler requirement for Pro*C/C++, Oracle Call Interface, Oracle C++ Call Interface, and Oracle XML Developer’s Kit (XDK) with Oracle Database 11g Release 2 (11.2):

IBM XL C/C++ Enterprise Edition for AIX, V9.0 April 2008 PTF:

You can download this software from the following link:

http://www-01.ibm.com/support/docview.wss?uid=swg24015075

Note:

Even if you do not install the IBM XL C/C++ compiler, you require the compiler for AIX Runtime Environment Component. The runtime environment file sets can be downloaded with no license requirements. The minimum recommended runtime environment for AIX 5.3 and AIX 6.1 is available at the following URL:IBM XL C/C++ for AIX, V10.1 Runtime Environment

http://www-01.ibm.com/support/docview.wss?rs=2239&uid=swg24019829

Abstract

This page contains the Runtime Environment package for use with applications developed using the IBM XL C/C++ for AIX, V10.1 compiler or its predecessors.

Download Description

If you wish to run an application developed using the IBM XL C/C++ for AIX, V10.1 compiler, but do not have the corresponding version of compiler or runtime already installed, then you need to install the Runtime Environment package.

Note: Depending on your application, in addition to the Runtime Environment package, you may also need to install the Utilities package. The Utilities package contains files that may be required by some third-party applications. Only download and install the Utilities package if instructed by your application provider.

Prerequisites

Operating system (one of):

IBM AIX V5.3 TL 5300-06, or higher, with bos.rte.libc 5.3.0.60, or higher
IBM AIX V6.1 with bos.rte.libc 6.0.0.0, or higher
IBM i 6.1 (i5/OS V6R1) PASE

Disk space: 45 MB

Installation Instructions

Refer to the README file for installation instructions.

Terms and Conditions
By downloading, installing, copying, accessing or using this software you agree to the terms and conditions outlined in the License Agreement. Please review the License Agreement and License Information documents prior to proceeding with download or installation.

Download package

The Runtime Environment package xlcpp.rte.10.1.0.aix.base.tar.Z contains the following files:
vacpp.cmp.rte
vacpp.memdbg.aix53.rte
vacpp.memdbg.rte
xlC.aix50
xlC.aix61
xlC.msg.Ja_JP
xlC.msg.en_US
xlC.msg.ja_JP
xlC.rte
xlsmp.aix53.rte
xlsmp.msg.EN_US.rte
xlsmp.msg.JA_JP.rte
xlsmp.msg.Ja_JP.rte
xlsmp.msg.ZH_CN.rte
xlsmp.msg.Zh_CN.rte
xlsmp.msg.en_US.rte
xlsmp.msg.ja_JP.rte
xlsmp.msg.zh_CN.rte
xlsmp.rte

Download	RELEASE DATE	LANGUAGE	SIZE(Bytes)	Download Options
xlcpp.rte.10.1.0.aix.base	2008-07-14	English	34310266	FTP

Technical support

The C++ Runtime Environment for AIX is not supported as a stand-alone software product.

Support for questions and problems relating to installation of the C++ Runtime Environment is available as part of AIX Software Maintenance (SWMA).

If you suspect a problem with the C++ Runtime Environment while using it with a third-party application, contact your application provider for support.

Full support (installation, defect support and general questions) for the C++ Runtime Environment is available with the software maintenance agreement for the XL C++ Compilers. A suitable test case is required for defect support; click here for details.

MustGather: No response (hang) on AIX 5.2 and higher versions produces no javacore

admin — Mon, 23 Apr 2012 20:20:14 +0000

Problem(Abstract)

MustGather for when your WebSphere Application Server is not responding (hangs) on an AIX 5.2 system and a “kill -3″ command does not produce javacore files. Gathering this information before calling IBM support will help familiarize you with the troubleshooting process and save you time.

Resolving the problem

If you already contacted support, continue to the component-specific MustGather information. Otherwise, click on the following link: MustGather: Read first for all WebSphere Application Server products.

No javacore generated specific MustGather information
The following steps will ensure that AIX is set up to accept a full core:

Go into SMIT and ensure that full core is enabled; or, from a command line, enter the following:
lsattr -Elsys0 | grep full

To set full core to true:

chdev -a fullcore=true -lsys0
Ensure that your ulimit -c and ulimit -f is set high enough.
Note: Units are in 512-byte blocks. Set to unlimited to ensure that you have enough free file system space in your install_root/bin directory (at least 750 MB to 2 GB).

To check the current settings run the following commands:

ulimit -c
ulimit -f

To set to unlimited, run the following commands:

ulimit -c unlimited
ulimit -f unlimited
Ensure that the WebSphere Application Server process can write to the install_root\bin directory (default, unless workingDirectory variable is changed).
Check the file, /etc/security/limits. The stanza for the user that runs the process should have fsize = -1 and core = -1. Setting these values to -1 changes the setting to unlimited; you must ensure that you have enough free space to handle a large core file.

Enter the following command:

gencore process_ID file_name

Example: To generate a core file, named core.1095, for the process with process ID 1095, enter the following command:

gencore 1095 core.1095

The gencore command creates a core file of the process specified by the ProcessID without terminating the process.

If you still do not get a core file after setting these parameters correctly, do the following:
1. Add the following lines to the beginning of the file, startupServer.sh:
  
  export DISABLE_JAVADUMP=true export IBM_NOSIGHANDLER=true
2. Restart all WebSphere Application Server java processes.
3. Enter the following command against the PID of the java process to test getting a core file:
  gencore process_ID file_name
If you can get a core from the test, wait for a hang to occur during an error condition.
If you are using WebSphere MQ, it might trap the signal and not propagate to the Java™ process.

Disable the WebSphere MQ signal handling by setting the following environment variable:

MQS_NO_SYNC_SIGNAL_HANDLING

Example:

export MQS_NO_SYNC_SIGNAL_HANDLING=true

When the Java process hangs, perform the following steps:

Enter the following command:
gencore process_ID core1.process_ID
Wait for 3 to 5 minutes to allow for core file generation.
Enter the following command:
gencore process_ID core2.process_ID
Wait for 3 to 5 minutes to allow for core file generation.
Enter the following command:
gencoreprocess_ID core3.process_ID
Enter the following command:
./getlib.sh java_executable_fullpath core_file_name

This creates getlib_archive.tar.Z.

Note: You must have dbx installed to run this command.
Enter the following command:
tar cvf core_file_name.tar core_file_name

This creates core_file_name.tar.

If the Developer Kit is 131 SR5 (131-20030618) or later, run the following command against each core file:

./jextract -o coreN.sdff [CORE_PATH]

Where,

`jextract`	is found in the `install_root/java/jre/bin` directory `coreN.sdff` is the output file and “N” is the number (for example: 1, 2, 3, and so on)
`[CORE_PATH]`	is the path and file name for the core file

Collect the following diagnostic information:

All dbxtrace.out files
All standard data file format (SDFF) files
errpt.out
All getlib_archive.tar.Z files
All core.tar files

All WebSphere Application Server log files for V6.0: 

Include all of the files from the following directory:

install_root/profiles/profile_name/logs/server_name

Include the following Application Server log files if they are located in a different location:

SystemOut
SystemErr
native_stdout
native_stderr

Send the server.xml file located in the following directory:

install_root/profiles/profile_name/configcells/cell_name/nodes/node_name/servers/server_name

All WebSphere Application Server log files for V5.1 and V5.0:

Include all of the files from the following directory:

install_root/logs/server_name

Include the following Application Server log files if they are located in a different location:

SystemOut
SystemErr
native_stdout
native_stderr

Send the server.xml file located in the following directory:

install_root/config/cells/cell_name/nodes/node_name/servers/server_name

Note: See the attached scripts dbxtrace_aix.sh and getlib.sh. To run the dbxtrace_aix.sh file, you must have the dbx component of AIX installed on the machine.

Follow instructions to send diagnostic information to IBM support

SEA Failover flapping on backup SEA

admin — Sat, 14 Apr 2012 20:16:52 +0000

Question

Why is the backup SEA adapter of my SEA failover flapping from Primary to Backup repeatedly?

Cause

The Shared Ethernet Adapter (SEA) failover hung or became unresponsive. The backup SEA adapter was flapping between Primary and Backup states which is seen as contention on the control channel between primary ( 1) and backup (2) of SEAs.

Answer

This issue can be caused by 2 different problems:

1) The backup SEA sends a pulse to the primary SEA to see if it is still alive. The primary VIO is not able to send heart beats to backup SEA fast enough due to a lack of available CPU cycles. The backup SEA with trunk priority 2 tries to become primary before it receives the reply and logs these SEAHA_PRIMARY, SEAHA_BACKUP errors. This can sometimes be resolved by changing the VIO CPUs from shared to dedicated.

Another resolution is to update the VIO servers to at least 2.2.0.12 FP24 SP02 to get the SEA fixes for this issue.

2) CPU folding enabled on VIO servers can cause SEA flapping and in turn will cause the VIO SEA to hang.

Processor folding: Processor folding currently is not supported for VIOS partitions. If processor folding is enabled on your VIOS, and migration media is used to move from VIOS 1.5 to 2.1.0.13 FP 23, or later, processor folding remains enabled. Upgrading via migration media does not change the processor folding state. If you have installed VIOS 2.1.3.0, or later, and have not changed the folding policy, then folding is disabled.

Check for CPU folding on VIOS:

$ oem_setup_env

# schedo -o vpm_fold_policy

If the value is anything other than 4, turn it off with this command:

# schedo -p -o vpm_fold_policy=4

The current value can also found in the ./kernel/kernel.snap file in the VIO snap.

Optimizing Sybase Adaptive Server Enterprise (ASE) for IBM AIX

admin — Mon, 09 Apr 2012 14:36:52 +0000

This is a paper from IBM and Sybase to assist database administrators (DBAs) and UNIX system administrators (SAs) working together. This white paper helps these professionals achieve the best possible performance from the Sybase Adaptive Server Enterprise (ASE) on IBM Power Systems running in the IBM AIX operating environment.

The focus of this white paper is on Sybase ASE 15.0.2 through ASE 15.5, running under AIX 5.3, 6.1, and 7.1 on the IBM POWER6 and IBM POWER7 processor-based servers. The recommendations of the original version of this white paper, published in March 2006, applied to IBM AIX 5L logical partitions (LPARs) with dedicated processors, memory, and devices. The 2009 revision of the paper also addressed all relevant aspects of IBM PowerVM including shared-processor LPARs (referred to as IBM Micro-Partitioning) and virtual I/O, features which are only available in the POWER 5, POWER6, and POWER7 processor-based servers with AIX 5.3 or later releases. The current revision of the white paper extends coverage to POWER7, enhancements to virtual I/O, and AIX 7.1 where relevant to ASE. Recent field and lab experience with POWER7 processor-based servers is applied to new or revised best practice recommendations. The recommendations apply to ASE Cluster Edition for AIX except where noted.

Full PDF: Optimizing Sybase Adaptive Server Enterprise(ASE) for IBM AIX

Upgrading CAA in a PowerHA v7.1 Cluster

admin — Sun, 08 Apr 2012 18:24:36 +0000

AIX 6100-06/7100-00 delivered the first version of the Operating System embedded
cluster technology also called as Cluster Aware AIX (CAA). AIX releases later to these
technology levels have modified CAA significantly. Hence updating to these later AIX
levels from 6100-06 or 7100-00 requires that CAA based cluster be stopped from
operation before the OS update. This document provides guidelines to customers for such
an OS update procedure.
Note that PowerHA SystemMirror 7.1 exploits the CAA technology in the AIX
Operating System and hence would require CAA cluster be stopped before the OS
updates.

CAA Migration Demo Resource Configuration

Upgrading CAA Overview Steps

Stop cluster services on all nodes
Remove CAA cluster via rmcluster command
Update to current TL/SP levels

– at least bos.cluster,bos.ahafs, and RSCT (CAAv2 no longer uses the soliddb)

– More commonly update entire TL/SP levels and it may require a reboot.

Update /etc/cluster/rhosts

– This generally should not be needed but early v2 release overwrites the file

Refresh -s clcomd
Synchronize the cluster

– This will recreate the CAA cluster

Restart cluster services on all nodes

Full details are provided in the following the following whitepaper

Upgrading CAA in a PowerHA v7.1 Cluster

Demo on Youtube:

SAS 9.2 on IBM AIX 7.1: Tuning guideline

admin — Sun, 08 Apr 2012 17:59:13 +0000

SAS 9.2 on IBM AIX 7.1:

Tuning guidelines

This paper provides a starting point for performance optimization from a system-wide perspective to create an enhanced environment for SAS 9.2 on IBM® POWER® processor-based servers that run IBM AIX® 7.1.

§ Technology levels:

Use the latest version of AIX 7 Version 7.1 with the latest maintenance level and fix pack. Refer to the following website: ibm.com/support/fixcentral

At publication of this paper, TL00 with fix pack is considered to be the latest technology level.

§ IBM Hardware Managements Console (HMC):

Keep the HMC and microcode up-to-date.

You can search update bulletins for HMC and microcode at:

www14.software.ibm.com/webapp/set2/subscriptions/pqvcmjd

§ VMM:

AIX 7.1 default VMM parameters are mostly tuned for optimal SAS performance, except for one parameter. If needed, you can change the other values by using the AIX vmo command. It is suggested that you start with the following parameter value:

nokilluid=10

minfree=use greater of 960 or (128 * number of logical processors)

maxfree=(minfree + j2_maxPageReadAhead * number of logical processors)

Note: The following have been designated as restricted values in AIX 7 and should not be changed unless asked by AIX support. These values are already in-line with what SAS recommends.

maxperm%
maxclient%
strict_maxclient
strict_maxperm

§ I/O:

Tune I/O at the filesystem layer by using the AIX ioo command to enable Enhanced Journaled File System (JFS2) to perform efficient caching. It is suggested that you start with the following parameter values:

j2_dynamicBufferPreallocation=256

j2_nBufferPerPagerDevice=2048 j2_maxPageReadAhead=1024 j2_minPageReadAhead=16

j2_nPagesPerWriteBehindCluster=64

For higher workloads, it is recommended to increase the value of j2_maxPageReadAhead upto 2048.

§ Network:

Tune network parameters by using the no command. If SAS applications (such as SPDS, SPDE and/or SAS/CONNECT) are being used, set the following parameter value:

tcp_nodelayack=1

§ Maximum user process:

If the maximum number of processes for a single user exceeds 2000, increase the value of maxuproc to prevent SAS processes from abnormal shutdown or delay.

Increase the maxuproc setting by using the AIX smit or chdev command, for example: chdev –l sys0 –a maxuproc=

§ User limits:

Increase user-process resource limits for SAS users and database instances as appropriate (for example, unlimited or some tuned value for all resources (Refer to the Note).

In the /etc/security/limits file, set -1 for all resources.

Note: The default AIX user-process resource limits might be too low for SAS power users or large enterprise-class deployments of SAS. When SAS processes end because of attempts to exceed these resource limits, the system administrator typically sets all user-process resource limits to unlimited (a numeric value of -1) for users who run SAS. The problem with this approach is that the increased multithreading and scalability support in newer versions of SAS, coupled with an unlimited setting for user-process resource limits, allows other users to potentially exhaust system resources such as processor, memory I/O, and paging space. Before increasing user-process resource limits, such as memory, to very high values, carefully consider the potential consequences.

§ Paging space:

Configure the paging space to include at least the following suggestions:

Place paging spaces on dedicated disks to eliminate I/O contention.

Use multiple paging spaces that are spread across multiple disks.

Make the primary paging space hd6 a little bigger than the secondary paging spaces.

Ensure that the paging space is sufficient to support the number of concurrent SAS processes (because the number of SAS processes can be dynamic, depending on application workload).

§ Volume groups (VGs):

Use the AIX Scalable or Big volume group.

The scalable VG implementation provides configuration flexibility with respect to the number of physical volumes (PVs) and logical volumes (LVs) that a given instance of the new VG type can accommodate. The configuration options allow any scalable VG to contain 32, 64, 128, 256, 512, 768, or 1024 disks and 256, 512, 1024, 2048, or 4096 LVs. You do not need to configure the maximum values of 1024 PVs and 4096 LVs when creating the VG to account for potential future growth. You can increase the initial settings later, as required.

§ Disk layout:

Minimize disk contention between SAS temporary space and data spaces.

Avoid disk contention by placing SAS temporary-space file systems and SAS data file systems on physically separate disks.

Use multiple storage-server controllers to further separate and isolate the I/O traffic between SAS temporary and data spaces.

Use multiple mount points for SAS file systems. Place the operating system, SAS, user, SAS temporary space and SAS data file systems on separate physical disks.

Consider creating multiple SAS WORK areas that can be used by groups of SAS users.

Create separate JFS2 log files on separate physical disks for each SAS file system.

Spread the I/O workload across many physical disk spindles rather than across fewer, larger-capacity disks. Determine the sizing, based on the quantity of disks rather than disk capacity. Do not wrap logical unit numbers (LUNs) around the same spindle sets.

Do not share disk spindles with a RDBMS.

§ Release-behind mechanism for JFS2:

This feature allows the file system to release the file pages from file system buffer cache as soon as an application has read or written the file pages.

This feature helps when the SAS application performs a great deal of sequential reads or writes and most often, once accessed, these file pages are not accessed again in the near future. This feature can be configured on a file system basis. When using the mount command, enable release-behind by specifying one of the three flags below:

Release-behind sequential read flag (-rbr),

Release-behind sequential write flag (-rbw),

Release-behind sequential read and write flag (-rbrw).

§ Host bus adapters (HBAs):

Use an adequate number of HBAs from storage to the host server to provide the required application bandwidth.

Consider high-performance storage channels, such as Fibre Channel technology instead of slower mediums.

If possible, use dynamic multipathing to spread the I/O load across multiple adapters.

§ Redundant Array of Independent Disks (RAID):

Implement storage system RAID striping across multiple physical disks.

Use RAID10 or RAID5, depending on the level of redundancy and total capacity instead of the usable capacity that is needed for each file system type.

Use Logical Volume Manager (LVM) striping instead of concatenation.

§ LVM striping:

When choosing the disk stripe or segment size, or array stripe size, note that AIX file systems are aligned on a 16 KB boundary.

A strip is the size of data to be written to each physical disk in the array. A stripe is the size of the full write across all the physical disks in the array. For example: strip size x number of disks = stripe size

The AIX LVM stripe size that you can select from the smit lv create panel is actually the single strip size (not stripe). In other words, it is the size of data to be written to each of the array disks; it is not the full stripe size across all the physical disks. Consider using an LVM stripe size of 64 K or 128 K. Stripe sizes of 256 K or 512 K has shown better I/O performance in the case of SAS 9.2 workloads.

Synchronize SAS BUFSIZE with the storage-system stripe size and the AIX LVM stripe size (if using LVM striping) and VMM read-ahead increments.

Synchronizing I/O sizes streamlines I/O processing and reduces the number of I/O requests to the storage subsystem.

§ Resources:

For detailed tuning information, refer to the following publications:

Best Practices for Configuring your IO Subsystem for SAS9 Applications http://support.sas.com/rnd/papers/sgf07/sgf2007-iosubsystem.pdf

How to Maintain Happy SAS Users http://support.sas.com/resources/papers/happyIT.pdf

AIX 6 best practices for SAS on IBM POWER6

http://www.sas.com

SAS on IBM AIX 6.1: Tuning guidelines

admin — Sun, 08 Apr 2012 17:26:12 +0000

SAS on IBM AIX 6.1:

Tuning guidelines

This paper provides a starting point for performance optimization from a system-wide perspective to create an enhanced environment for SAS 9 on IBM® POWER® processor-based servers that run IBM AIX® 6.1.

§ Technology levels:

Use the latest version of AIX 6 Version 6.1 with the latest maintenance level and fix pack. Refer to the following website: ibm.com/support/fixcentral

At publication of this paper, TL03 with fix pack is considered to be the latest technology level.

§ IBM Hardware Managements Console (HMC):

Keep the HMC and microcode up-to-date.

You can search update bulletins for HMC and microcode at:

www14.software.ibm.com/webapp/set2/subscriptions/pqvcmjd

§ VMM:

AIX 6.1 default VMM parameters are mostly tuned for optimal SAS performance, except for one parameter. If needed, you can change the other values by using the AIX vmo command. It is suggested that you start with the following parameter value:

nokilluid=10

minfree=use greater of 960 or (128 * number of logical processors)

maxfree=(minfree + j2_maxPageReadAhead * number of logical processors)

Note: The following have been designated as restricted values in AIX 6 and should not be changed unless asked by AIX support. These values are already in-line with what SAS recommends.

lru_file_repage
minperm%
maxperm%
maxclient%
strict_maxclient
strict_maxperm

§ I/O:

Tune I/O at the filesystem layer by using the AIX ioo command to enable Enhanced Journaled File System (JFS2) to perform efficient caching.

It is suggested that you start with the following parameter values:

j2_dynamicBufferPreallocation=256

j2_nBufferPerPagerDevice=2048 j2_maxPageReadAhead=1024 j2_minPageReadAhead=16

j2_nPagesPerWriteBehindCluster=64

For higher workloads, it is recommended to increase the value of the j2_maxPageReadAhead parameter upto 2048.

§ Network:

Tune network parameters by using the no command. If SAS applications (such as SPDS, SPDE and/or SAS/CONNECT) are being used, set the following parameter value:

tcp_nodelayack=1

§ Maximum user process:

If the maximum number of processes for a single user exceeds 2000, increase the value of maxuproc to prevent SAS processes from abnormal shutdown or delay.

Increase the maxuproc setting by using the AIX smit or chdev command, for example: chdev –l sys0 –a maxuproc=

§ User limits:

Increase user-process resource limits for SAS users and database instances as appropriate (for example, unlimited or some tuned value for all resources (Refer to the Note).

In the /etc/security/limits file, set -1 for all resources.

§ Paging space:

Configure the paging space to include at least the following suggestions:

Place paging spaces on dedicated disks to eliminate I/O contention.

Use multiple paging spaces that are spread across multiple disks.

Make the primary paging space hd6 a little bigger than the secondary paging spaces.

Ensure that the paging space is sufficient to support the number of concurrent SAS processes (because the number of SAS processes can be dynamic, depending on application workload).

§ Volume groups (VGs):

Use the AIX Scalable or Big volume group.

§ Disk layout:

Minimize disk contention between SAS temporary space and data spaces.

Avoid disk contention by placing SAS temporary-space file systems and SAS data file systems on physically separate disks.

Use multiple storage-server controllers to further separate and isolate the I/O traffic between SAS temporary and data spaces.

Use multiple mount points for SAS file systems. Place the operating system, SAS, user, SAS temporary space, and SAS data file systems on separate physical disks.

Consider creating multiple SAS WORK areas that can be used by groups of SAS users.

Create separate JFS2 log files on separate physical disks for each SAS file system.

Spread the I/O workload across many physical disk spindles rather than across fewer, larger-capacity disks. Determine the sizing based on the quantity of disks rather than disk capacity. Do not wrap logical unit numbers (LUNs) around the same spindle sets.

Do not share disk spindles with a RDBMS.

§ Release-behind mechanism for JFS2:

This feature allows the file system to release the file pages from file system buffer cache as soon as

an application has read or written the file pages. This feature helps when the SAS application performs a great deal of sequential reads or writes and most often, once accessed, these file pages are not accessed again in the near future. This feature can be configured on a file system basis. When using the mount command, enable release-behind by specifying one of the following three flags:

Release-behind sequential read flag (-rbr),

Release-behind sequential write flag (-rbw),

Release-behind sequential read and write flag (-rbrw).

§ Host bus adapters (HBAs):

Use an adequate number of HBAs from storage to the host server to provide the required application bandwidth.

Consider high-performance storage channels, such as Fibre Channel technology instead of slower mediums.

If possible, use dynamic multipathing to spread the I/O load across multiple adapters.

§ Redundant Array of Independent Disks (RAID):

Implement storage system RAID striping across multiple physical disks.

Use RAID10 or RAID5, depending on the level of redundancy and total capacity instead of the usable capacity that is needed for each file-system type.

Use Logical Volume Manager (LVM) striping instead of concatenation.

§ LVM striping:

When choosing the disk stripe or segment size, or array stripe size, note that AIX file systems are aligned on a 16 KB boundary.

The AIX LVM stripe size that you can select from the smit lv create panel is actually the single strip size (not stripe). In other words, it is the size of data to be written to each of the array disks; it is not the full stripe size across all the physical disks. Consider using an LVM stripe size of 64K or 128 K. Stripe sizes of 256 K or 512 K has shown better I/O performance in the case of SAS 9.2 workloads.

Synchronize SAS BUFSIZE with the storage-system stripe size and the AIX LVM stripe size (if using LVM striping) and VMM read-ahead increments.

Synchronizing I/O sizes streamlines I/O processing and reduces the number of I/O requests to the storage subsystem.

§ Resources:

For detailed tuning information, refer to the following publications:

Best Practices for Configuring your IO Subsystem for SAS9 Applications http://support.sas.com/rnd/papers/sgf07/sgf2007-iosubsystem.pdf

How to Maintain Happy SAS Users http://support.sas.com/resources/papers/happyIT.pdf

AIX 6 best practices for SAS on IBM POWER6

http://www.sas.com/partners/directory/ibm/AIXBestPractice.pdf

SAS AIX 5L Tuning Guide

admin — Sun, 08 Apr 2012 15:36:28 +0000

§ Technology levels:

Use the latest version of IBM AIX 5L™ Version 5.3 with the latest maintenance level and fix pack. Refer to the following website: ibm.com/support/fixcentral

At publication of this paper, TL10 with e-fix is considered to be the latest technology level.

§ IBM Hardware Managements Console (HMC):

Keep the HMC and microcode up-to-date.

www14.software.ibm.com/webapp/set2/subscriptions/pqvcmjd

§ VMM:

Tune VMM by using the AIX vmo command to enable Enhanced Journaled File System (JFS2) to perform efficient caching. It is suggested that you start with the following parameter values:

maxclient%=80

maxperm%=80

minperm%=5

lru_file_repage=0

strict_maxperm=0

minfree=128 * number of logical processors

maxfree=minfree + j2_maxPageReadAhead * number of logical processors

nokilluid=10

It is also suggested to set the following parameter value to 1 from the default value of 0.

page_steal_method = 1

§ I/O:

Tune I/O at the file system layer by using the AIX ioo command to enable JFS2 to perform efficient caching. It is suggested that you start with the following parameter values:

j2_dynamicBufferPreallocation=256

j2_maxPageReadAhead=128

j2_minPageReadAhead=8

§ Network:

Tune network parameters by using the no command. If SAS applications (such as SPDS, SPDE and/or SAS/CONNECT) are being used, set the following parameter value:

tcp_nodelayack=1

§ Maximum user process:

If you anticipate that the maximum number of processes for a single user will exceed 2000, increase the value of maxuproc to prevent abnormal delay or shutdown of SAS processes.

Increase the maxuproc setting by using the AIX smit or chdev command. For example: chdev –l sys0 –a maxuproc=

§ User limits: Increase the AIX user-process resource limits for SAS users and database instances, as appropriate (for example, unlimited or some tuned value for all resources (Refer to the Note).

In the /etc/security/limits file, set -1 for all resources.

Note: The default user-process resource limits might be too low for SAS power users or large enterprise-class SAS deployments. When SAS processes end because of attempts to exceed these resource limits, the system administrator typically sets all user-process resource limits to unlimited (a numeric value of -1) for users who run SAS. The problem with this approach is that the increased multithreading and scalability support in newer versions of SAS, coupled with an unlimited setting for user-process resource limits, allows other users to potentially exhaust system resources such as processor, memory, I/O and paging space. Before increasing user-process resource limits, such as memory, to very high values, carefully consider the potential consequences.

§ Paging space:

Configure the paging space to include at least with the following suggestions:

Place paging spaces on dedicated disks to eliminate I/O contention.

Use multiple paging spaces that are spread across multiple disks.

Make the primary paging space hd6 a little bigger than the secondary paging spaces.

Ensure sufficient paging space to support all concurrent SAS processes (because the number of SAS processes can be dynamic, depending on application workload).

§ Volume groups (VGs):

Use the AIX Scalable or Big volume group.

The scalable VG implementation provides configuration flexibility with respect to the number of physical volumes (PVs) and logical volumes (LVs) that a given instance of the new VG type can accommodate. The configuration options allow any scalable VG to contain 32, 64, 128, 256, 512, 768 or 1024 disks and 256, 512, 1024, 2048 or 4096 LVs. You do not need to configure the maximum values of 1024 PVs and 4096 LVs when creating the VG to account for potential future growth. You can increase the initial settings later, as required.

§ Disk layout:

Minimize disk contention between SAS temporary space and data spaces.

Avoid disk contention by placing SAS temporary-space file systems and SAS data file systems on physically separate disks.

Use multiple storage-server controllers to further separate and isolate the I/O traffic between SAS temporary and data spaces.

Use multiple mount points for SAS file systems. Place the operating system, SAS, user, SAS temporary space and SAS data file systems on separate physical disks.

Consider creating multiple SAS WORK areas that can be used by groups of SAS users.

Create separate JFS2 log files on separate physical disks for each SAS file system.

Do not share disk spindles with a RDBMS.

§ Host bus adapters (HBAs):

Use an adequate number of HBAs from storage to the host server to provide the required application bandwidth.

Consider high-performance storage channels, such as Fibre Channel technology instead of slower mediums.

If possible, use dynamic multipathing to spread the I/O load across multiple adapters.

§ Release-behind mechanism for JFS2:

This feature allows the file system to release the file pages from file system buffer cache as soon as an application has read or written the file pages. This feature helps when the SAS application performs a great deal of sequential reads or writes and most often, once accessed, these file pages are not accessed again in the near future. This feature can be configured on a file system basis. When using the mount command, enable release-behind by specifying one of the following three flags:

Release-behind sequential read flag (-rbr),

Release-behind sequential write flag (-rbw),

Release-behind sequential read and write flag (-rbrw).

§ RAID:

Implement storage system RAID striping across multiple physical disks.

Use RAID10 or RAID5, depending on the level of redundancy and total capacity instead of the usable capacity that is needed for each file-system type.

Use LVM striping instead of concatenation.

§ LVM striping:

When choosing the disk stripe or segment size, or array stripe size, note that AIX file systems are aligned on a 16 KB boundary.

Synchronize SAS BUFSIZE with the storage-system stripe size and the AIX LVM stripe size (if using LVM striping) and VMM read-ahead increments.

Synchronizing I/O sizes streamlines I/O processing and reduces the number of I/O requests to the storage subsystem.

§ AIX 5L thread tuning: To improve SAS application performance, set the following parameter: AIXTHREAD_SCOPE=S

Running SAS applications on Virtual I/O based storage configuration

admin — Sun, 08 Apr 2012 15:24:58 +0000

Previous SAS benchmarks relied heavily on directly-attached storage for high disk I/O throughput capability. This white paper shows that VIOS-attached disks can meet the same high I/O throughput needs as the directly-attached storage without compromising performance. The white paper compares two similar configurations running the same benchmark; one configuration uses directly- attached disks and the other uses VIOS-attached disks to handle high I/O activity needed by the SAS computational back-end servers.

Introduction

With the introduction of IBM® Dynamic Infrastructure initiative, more customer production systems are using virtualization features available through IBM PowerVM™. Today’s modern data center has embraced virtualization and is now a key component of their architecture. Features such as micro-partitioning, virtualized Ethernet and storage help to deliver superior business and IT services with agility and speed. Customers realize that consolidating resources through virtualization increases utilization beyond what they used to be five years ago. Not only does virtualization decrease total cost of ownership, it also helps to improve speed of service delivery and flexibility.

In previous SAS benchmarks performed through the collaboration of SAS and IBM engineers, virtualization features proved to increase utilization by accommodating more SAS users compared to using dedicated servers. In previous benchmarks, the use of virtual storage was limited to the system filesystems (rootvg) where little I/O was needed. This paper studies the effect of using virtual storage when high I/O SAS filesystems such as /saswork and /sasdata are accessed through the Virtual I/O Server (VIOS). The test methodology mentioned in this paper is the same as the one mentioned in the paper entitled Deploying SAS Enterprise Business Intelligence 9.2 on IBM PowerVM.

The SAS test environment

The SAS environment is a three-tier architecture, comprising a Web and application tier, a services middle tier, and a computational back-end tier. Using PowerVM, the different tiers are deployed in separate logical partitions (LPARs) within the same physical server. There are two SAS test environment configurations used. The first configuration, known as the baseline configuration, uses VIOS-attached disks for system filesystems only and high I/O disks are attached directly to the SAS computational back-end server. The second configuration is where the high I/O disks are reconfigured to be on the VIOS and shared back to the computational back-end server.

Full Word Document: SAS_vios_testing_final
Full PDF Document: SAS_vios_testing_final

Live Partition Mobility for Oracle RAC

admin — Sun, 08 Apr 2012 15:16:46 +0000

This paper documents the concept and recommendation of using Live Partition Mobility (LPM) in an Oracle RAC environment. It describes the configuration of the IBM Power systems infrastructure and Oracle RAC to perform the Live Partition Mobility of an Oracle RAC node.

The paper describes two scenarios, the first is using LPM and show you how to setup the configuration and let you understand the interaction of all components.

The second scenario stops Oracle cluster on the node to migrate before the LPM operation.

This paper is illustrated with a real example of the LPM of a RAC node from a Porwer6 processor based source server to a Power7 target server.

Full PDF: http://www.aixmind.com/wp-content/uploads/2012/04/Live-Partition-Mobility-for-Oracle-RAC.pdf

How to Apply ACL to New Files Automatically

admin — Sun, 08 Apr 2012 14:53:14 +0000

Question

How can I set up a directory that will automatically apply ACLs (Access Control Lists)?

Answer

Using an ACL is a way to set restrictions on the use of a file beyond the standard AIX or Unix discretionary access control. These can be set on an individual or group basis, and rights can be given or taken away as needed.

The concept of having a directory with ACLs on it, and then automatically applying those ACLs to any files created in that directory is called inheritance. AIX ACLs do not support inheritance, but NFSV4 ACLs do. NFSV4 ACL is platform-independent, so it can be supported on many clients or servers.

Most filesystem types in AIX do not support NFSV4 ACLs. CFS, UDF, JFS, and JFS2 with extended attribute version 1 (EAv1) do not support these. However JFS2 filesystems using EAv2 do, so they can be used for this purpose.

1. Use a JFS2 filesystem
If you have an existing JFS2 filesystem that can be used, or create one if you do not have one already

# crfs -v jfs2 -g datavg -m /myfs -A yes -a size=2G

2. Change the filesystem to use Extended Attributes Version 2

# chfs -a ea=v2 /myfs

3. Create a directory, or identify an existing directory

# cd /myfs
# mkdir newdir

4. Convert the directory to use NFS4 ACLs:

# aclconvert -t NFS4 newdir

5. Edit the ACL

# export EDITOR=/usr/bin/vi
# acledit newdir

You should see this in the file:

*
* ACL_type NFS4
*
*
* Owner: root
* Group: system
*
s:(OWNER@): a rwpRWxDaAdcCs
s:(OWNER@): d o
s:(GROUP@): a rRxadcs
s:(GROUP@): d wpWDACo
s:(EVERYONE@): a rRxadcs
s:(EVERYONE@): d wpWDACo

To allow inheritance for all files and directories underneath this dir, add the strings “fi” (for files) and “di” (for directories) to any ACLs you create. Those ACLs will be propagated to each file created from now on.

*
* ACL_type NFS4
*
*
* Owner: root
* Group: system
*
s:(OWNER@): a rwpRWxDaAdcCs fidi
s:(OWNER@): d o
s:(GROUP@): a rRxadcs
s:(GROUP@): d wpWDACo
s:(EVERYONE@): a rRxadcs
s:(EVERYONE@): d wpWDACo

Create a file in your directory and check the ACL list on it:

# cd newdir
# touch newfile
# aclget newfile

*
* ACL_type NFS4
*
*
* Owner: root
* Group: system
*
s:(OWNER@): a rwpRWxDaAdcCs fidi

Who’s using my port? (AIX, Linux, Windows example)

admin — Tue, 03 Apr 2012 03:58:58 +0000

Question

When an instance is started (db2start), listeners are started to accept the connect request from clients. Sometimes, the listeners may fail to start because the port is being used by a different application. This causes error: SQL5043N. To find which application is using the port, run netstat followed by rmsock on AIX. On Linux and Windows, netstat alone will do.

Cause

The db2start command gives a warning, SQL5043N, “Support for one or more communications protocols failed to start successfully. However, core database manager functionality started successfully.” The following message is logged in the db2diag.log file.
2007-01-23-22.45.33.315991-300 LEVEL: Error
PID : 3268824 TID : 1 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, common communication, sqlcctcpconnmgr, probe:46
MESSAGE : ADM7007E The SVCENAME DBM configuration parameter, "50000", is configured with a port or a service name. When it is configured with a service name, the TCP/IP services files is used to map the service name to a port number. The port specified in this field is being used by another process. Resolve this problem by either deleting the process using the port or use another port.

The db2diag.log file shows that the TCP/IP listener (db2tcpcm) failed to start because the required port, 50000 in this example, was already being used by another application. The simple resolution is to use a different port for listener. If you must use the port 50000, you should terminate the application that’s using it.

The commands below demonstrate how to determine which application or process is using the port. Please note that SQL5043N can be returned for various reasons. This technote only applies to the case where the port is used by another application.

Answer

AIX Command
1. netstat -Aan | grep
- This shows if the specified is being used. The hex number in the first column is the address of protocol control block (PCB)

2. rmsock tcpcb
- This shows the process who is holding the socket. Note: This command must be run as root..

AIX Example

Let’s set SVCENAME to 30542, so that the listener will use this port. Then, use the commands above to check if the port is indeed being used by DB2 LUW.

$ db2 update dbm cfg using svcename 30542
$ db2start
$ netstat -Aan | grep 30542

f10000f303321b58 tcp4 0 0 *.30542 *.* LISTEN

The netstat command above shows that the port 30542 is being used for listening. To confirm that DB2 LUW is using the port, run rmsock as root like following below:

$ rmsock f10000f303321b58 tcpcb
The socket 0x3321800 is being held by proccess 692476 (db2sysc).

The rmsock command shows it’s db2sysc process that’s using the port and its PID is 692476. So, in this case, no one else but DB2 is using this port.

Note that rmsock, unlike what its name implies, does not remove the socket, if the socket is being used by any process. Instead of removing the socket, it just reports the process holding the socket. Also note that the second argument of rmsock is the protocol. It’s tcpcb in the example to indicate that the protocol is TCP.

Windows Command

1. netstat -aon | findstr “”
- This shows if the specified is being used. The number in the last column is the process id (PID) of the process holding the socket. Once PID is determined, one can refer to “Windows Task Manager” to determine which application corresponds to the PID.

Windows Example

C:\>netstat -aon | findstr "50000"
TCP 0.0.0.0:50000 0.0.0.0:0 LISTENING 2564

C:\>pslist 2564

Process information for MACHINENAME:
Process information for MACHINENAME:

Name Pid Pri Thd Hnd Priv CPU Time Elapsed Time
db2syscs 2564 8 15 366 30912 0:00:02.859 2:12:08.564

The example above shows the use of pslist to determine the name of the process. Note that pslist is a free command available from Microsoft Sysinternals at http://www.microsoft.com/technet/sysinternals/default.mspx .

Linux Command

1. netstat -anp | grep
- This shows the PID and the program name that uses the port. The command must be run as root.

2. Alternatively, one can also run
fuser -n tcp

Linux Example

Suppose the port 12345 is being used by someone else. Find out who by running the following.

# netstat -anp | grep 12345
# netstat -anp | grep 12345
tcp 0 0 127.0.0.1:12345 0.0.0.0:* LISTEN 6629/ssh
tcp 0 0 ::1:12345 :::* LISTEN 6629/ssh

ssh with the PID 6629 is using the port. Find more info about it.

# ps -efl | grep 6629
4 S root 6629 29716 0 75 0 - 6976 - 14:05 pts/4 00:00:00 ssh testserver -D 12345 -l db2inst1
0 S root 7648 7302 0 78 0 - 742 pipe_w 14:07 pts/7 00:00:00 grep 6629

In this case, the user db2inst1 is deliberately using the port 12345 by specifying -D option of ssh.

Setting up your DB2 High Availability (HA) environment on Power7 systems

admin — Mon, 02 Apr 2012 20:18:24 +0000

Abstract

This technote provides information and the manual steps required to install new DB2 products or existing DB2 products on Power7 systems running the following minimum supported operation systems levels: AIX® 5.3 TL9, AIX 6.1 TL4, or AIX 7.1. This technote is applicable to scenarios where the DB2 High Availability (HA) Feature is used.

Content

Before you begin

For a list of the DB2 fix pack levels that automatically support Power7 systems, see:
http://www-01.ibm.com/support/docview.wss?uid=swg21420730

If you can not upgrade to a DB2 fix pack level that supports Power7 and you are using the DB2 High Availbility Feature and Tivoli System Multiplatform (SA MP) products, follow the steps in this technote to upgrade SA MP to a level supported on the POWER7® hardware.

Overview

Step #1 – Install the appropriate level of JDK.
Step #2 – Select one of the following scenarios that applies to your environment :

Note: On AIX 5.3 TL11 systems and above, SA MP 3.2.0.0 is the minimum level with a supported RSCT version. If this is a new DB2 installation, follow scenario (a) using the SA MP 3.2.0.0 package. If you already have an existing DB2 integrated High Availability environment with an older SA MP level, follow scenario (c) to migrate to SA MP 3.2.0.0.

a) New installation for DB2 V9.5, DB2 V9.7, or DB2 V9.8

b) Migration from SA MP 3.1.x / RSCT 2.5.4.0 to SA MP 3.1.0.6 special package with RSCT 2.5.4.2
(typically describes environments with DB2 V9.7 or DB2 V9.8. Also applies to DB2 V9.5 environments using SA MP 3.1.x already)

c) Migration scenario from SA MP 2.2..x / RSCT 2.5.4.0 to SA MP 3.1.0.6 special package with RSCT 2.5.4.2 (typically describes environments with DB2 V9.5 Fix Pack 5 and earlier fix packs)

Step #1 – JDK installation
For each platform and DB2 release combination, install the appropriate level of JDK:

Set the initial configuration on all nodes. On all nodes, set the following environment variable:
# export JAVA_COMPILER=noneInstall the appropriate DB2 release.

Set the initial configuration on all nodes. On all nodes, set the following environment variable to re-enable the Just-In-Time (JIT) compiler:
# export JAVA_COMPILER=

Download and install the appropriate JDK package for the DB2 release level:

DB2 release	JDK AIX	JDK Linux
V91 FP8	JDK 5 SR 11 http://www.ibm.com/developerworks/java/jdk/aix/j564/fixinfo.html	https: //www14.software.ibm.com/webapp/iwm/web/preLogin.do?source= sdk5&S_PKG=ipseries645sr11&S_TACT=105AGX05&S _CMP=JDK
V95 FP5
V97 FP1	JDK 6 SR 7 http://www.ibm.com/developerworks/java/jdk/aix/j664/Java6_64.fixinfo.html	https: //www14.software.ibm.com/webapp/iwm/web/preLogin.do?source= swg-sdk6&S_PKG=ipseries64_6sr7&S_TACT=105AGX05 &S_CMP=JDK
V98 GA

Update the JDK_PATH database manager configuration parameter to point to the top directory of the new JDK. For example, if the new JDK is installed in the /home/user1/bin/jdk64 path, under which you should see directories like bin, jre, and lib, then run the following:
db2 update dbm cfg using JDK_PATH /home/user1/bin/jdk64For Version 9.7 and Version 9.8, update DB2LIBPATH. For example, if the new JDK is installed in the /home/user1/bin/jdk64 path, under which you should see directories like bin, jre, and lib, then run the following command:
set DB2LIBPATH=/home/usr1/bin/jdk64/jre/lib/ppc64

Restart the DB2 instance to make use of the new JDK. This step should be run because the JDK shipped in existing DB2 fix packs will not work otherwise.
db2stop force
db2start

If you apply a DB2 fixpak that supports Power7 (which includes a JRE that supports Power7), (Refer to this techNote http://www-01.ibm.com/support/docview.wss?uid=swg21420730), you can reset the JDK_PATH parameter to its original value.

Step #2 (a) – New installation
All steps should be run as root unless indicated otherwise. References to the 3.1.0.6 package should be substituted for the 3.2.0.0 package on AIX 5.3 TL11 and above, and for the 3.2.0.1 package on the AIX 7.1 operating systems.

Install the JDK as described in the previous section
Set the initial configuration on all nodes. On all nodes, set the following environment variables:
# export CT_MANAGEMENT_SCOPE=2
# export JAVA_COMPILER=none

Install the DB2 product and the SA MP product.

V9.5

V9.7 and V9.8

Install the DB2 product without the integrated SA MP:
# cd
# ./db2_install -f NOTSAMP

Install the SA MP 3.1.0.6 special package:
# cd
# ./installSAM
Accept license agreement to proceed with the installation.

After the installation, an integrated HA installation will copy the SA MP policies scripts to the correct location. Since SA MP is installed separately in this case, run the following command to copy the scripts over:
# /opt/IBM/db2/V9.5/install/tsamp/db2cptsa

Install the DB2 product with the integrated SA MP:
# cd
# ./db2_install

Install the SA MP 3.1.0.6 special package:
# cd
# ./installSAM

Step #2 (b) – Migration from SA MP 3.1.X stream
All steps should be run as root unless indicated otherwise. References to the 3.1.0.6 package should be substituted for the 3.2.0.0 package on AIX 5.3 TL11 and above, and for the 3.2.0.1 package on the AIX 7.1 operating systems.

Set the initial configuration on all nodes. On all nodes, set the following environment variables:
# export CT_MANAGEMENT_SCOPE=2
# export JAVA_COMPILER=none
Stop the DB2 instance. As the instance owner, run the following command:
# db2stop force
You should receive the following message after running the db2stop command:
SQL1064N DB2STOP processing was successful.
Stop the existing cluster domain using the following command:
# stoprpdomain –f
Install the SA MP 3.1.0.6 special package by running the following commands:
# cd
# ./installSAM
Start the existing cluster domain using the following command:
# startrpdomain
Ensure that the domain is started and that all the nodes in the domain are online by running the following commands:
# lsrpdomain
Name OpState RSCTActiveVersion MixedVersions TSPort GSPort
ha Online 2.5.4.0 No 12347 12348

# lsrpnode
Name OpState RSCTVersion
el9-89-205 Online 2.5.4.2
el9-89-206 Online 2.5.4.2

Post-migration steps:

Verify the active and installed version number by running the following command:
# lssrc -ls IBM.RecoveryRM

Daemon State:
My Node Name : el9-89-205
Master Node Name : el9-89-205 (node number = 1)
Our IVN : 3.1.0.6
Our AVN : 3.1.0.1

# lsrpdomain
Name OpState RSCTActiveVersion MixedVersions TSPort GSPort
ha Online 2.5.4.0 No 12347 12348

# lsrpnode
Name OpState RSCTVersion
el9-89-205 Online 2.5.4.2
el9-89-206 Online 2.5.4.2
Update the RSCTActiveVersion. Ensure that all the nodes are online before issuing the following command:
# runact -c IBM.PeerDomain CompleteMigration Options=0
Resource Class Action Response for CompleteMigration

# lsrpdomain
Name OpState RSCTActiveVersion MixedVersions TSPort GSPort
ha Online 2.5.4.2 No 12347 12348
Activate the new features and finish the migration by running the following commands:
# samctrl –m

# lssrc -ls IBM.RecoveryRM

Daemon State:
My Node Name : el9-89-205
Master Node Name : el9-89-205 (node number = 1)
Our IVN : 3.1.0.6
Our AVN : 3.1.0.6

# lssam
Online IBM.ResourceGroup:db2_db2inst1_0-rg Nominal=Online
‘- Online IBM.Application:db2_db2inst1_0-rs
|- Online IBM.Application:db2_db2inst1_0-rs:el9-89-205
‘- Offline IBM.Application:db2_db2inst1_0-rs:el9-89-206
Online IBM.Equivalency:db2_db2inst1_0-rg_group-equ
|- Online IBM.PeerNode:el9-89-205:el9-89-205
‘- Online IBM.PeerNode:el9-89-206:el9-89-206

Step #2 (c) – Migration from SA MP 2.2.X stream
All steps should be run as root unless indicated otherwise. References to the 3.1.0.6 package should be substituted for the 3.2.0.0 package on AIX 5.3 TL11 and above, and for the 3.2.0.1 package on the AIX 7.1 operating systems.

Set the initial configuration on all nodes. On all nodes, set the following environment variables:
# export CT_MANAGEMENT_SCOPE=2
# export JAVA_COMPILER=none
As instance owner, stop the DB2 instance by running the following command:
# db2stop force
Stop the existing cluster domain by running the following command:
# stoprpdomain –f
Install the SA MP 3.1.0.6 special package by running the following commands:
# cd
# ./installSAM
Install the appropriate SA MP product license file by running the following commands:
# samlicm –i
# samlicm -s
Product: IBM Tivoli System Automation for Multiplatforms 3.1
Product ID: 101
Product Annotation:
Creation date: Sun Mar 19 18:00:00 2006
Expiration date: Thu Dec 31 17:59:59 2037
Start the existing cluster domain by running the following commands:
# startrpdomain

Post-migration steps:

Verify the active and installed version number by running the following commands:
# lssrc -ls IBM.RecoveryRM

Daemon State:
My Node Name : el9-89-205
Master Node Name : el9-89-205 (node number = 1)
Our IVN : 3.1.0.6
Our AVN : 2.2.5.7

# lsrpdomain
Name OpState RSCTActiveVersion MixedVersions TSPort GSPort
ha Online 2.5.4.0 No 12347 12348

# lsrpnode
Name OpState RSCTVersion
el9-89-205 Online 2.5.4.2
el9-89-206 Online 2.5.4.2
Update the RSCTActiveVersion. Ensure that all the nodes are online before issuing the following command:
# runact -c IBM.PeerDomain CompleteMigration Options=0
Resource Class Action Response for CompleteMigration

# lsrpdomain
Name OpState RSCTActiveVersion MixedVersions TSPort GSPort
ha Online 2.5.4.2 No 12347 12348
Activate the new features and finish the migration by running the following commands:
# samctrl –m

# lssrc -ls IBM.RecoveryRM

Daemon State:
My Node Name : el9-89-205
Master Node Name : el9-89-205 (node number = 1)
Our IVN : 3.1.0.6
Our AVN : 3.1.0.6

# lssam
Online IBM.ResourceGroup:db2_db2inst1_0-rg Nominal=Online
‘- Online IBM.Application:db2_db2inst1_0-rs
|- Online IBM.Application:db2_db2inst1_0-rs:el9-89-205
‘- Offline IBM.Application:db2_db2inst1_0-rs:el9-89-206
Online IBM.Equivalency:db2_db2inst1_0-rg_group-equ
|- Online IBM.PeerNode:el9-89-205:el9-89-205
‘- Online IBM.PeerNode:el9-89-206:el9-89-206

Frequently Asked Questions
Where can I get RSCT 2.5.4.0?
After upgrading to AIX 6.1 TL4, RSCT will be moved to RSCT 2.5.4.0. The full TL4 can be downloaded here: http://www-933.ibm.com/eserver/support/fixes/fixcentral/pfixpacks/61

Why do we need a SA MP 3.1.0.6 special package?
SA MP 2.2.x is not compatible with RSCT 2.5.4.2. Therefore, you must upgrade SA MP to the 3.1.x release level. If you apply Version 9.5 Fix Pack 6a (or a later fix pack), V9.7 Fix Pack 2 (or a later fix pack), or V9.8 Fix Pack 2 (or a later fix pack) , the RSCT level is automatically updated to the correct level. On AIX 5.3 TL11 and above systems, SA MP 3.2.0.0 is the minimum level with a supported RSCT version.

Where can I get a SA MP 3.1.0.6 special package?
Contact IBM support.

Where can I get a SA MP 3.1.x or 3.2.x license?
To obtain a license for Tivoli SA MP 3.1 or 3.2:
1. Go to the Passport Advantage site: http://www.ibm.com/software/howtobuy/passportadvantage/
2. Download a refreshed DB2 Enterprise Server Edition image or a High Availability feature image.
3. Each of these images contain a license key for the Tivoli SA MP product.
4. Place the license key under the SA MP installation directory’s license folder to proceed with SA MP installation.

During SA MP 3.1.0.6 special package installation, the following message is displayed. What does it mean?
/opt/IBM/tsamp/sam/bin/samadapter[51]: 9-SSU: bad number

This message indicates an installation problem that will be addressed in a future SA MP release version. You can continue using SA MP since samadapter is not used by the DB2 product.

During DB2 installation, the following message is displayed: “WARNING: A minor error occurred while installing”. However, there are no errors in the installation log.
This warning might be due to an ITMA installation failure. Check for javacore.* under the /tmp folder referencing ITMInstall. ITMA is not supported on Power7 systems but the DB2 installer tries to install it anyway. This defect will be addressed in APAR IC69770.

Where can I get more details on SA MP installation and migration?
Tivoli SA MP 3.1 – Installation and Configuration Guide
http://publib.boulder.ibm.com/tividd/td/IBMTivoliSystemAutomationforMultiplatforms3.1.html
Tivoli SA MP 3.2 – Installation and Configuration Guide
http://publib.boulder.ibm.com/tividd/td/IBMTivoliSystemAutomationforMultiplatforms3.2_or_later.html

The lssam command outputs for non-root users do not seem correct as some resources are missing, Is there a problem with the installation?
This issue is an expected behavior for RSCT 2.5.4.0 level and above. It applies to all non-root IDs and to all resource classes and their resources. The non-root user must have appropriate ACL permission on any cluster node remote from where the query is executed.

Do DB2 products on the Linux® operating system support Power7 systems?
Yes. No special setup is required on that operating system. http://www-01.ibm.com/support/docview.wss?uid=swg21420730

After installing SA MP 3.2.x, the lssam output displays the following error messages. What does it mean?

(lsrsrc-api) 2610-440 Permission is denied to access a resource specified in this command.\nNetwork Identity regress@0x75bd611f6bab1867/clusteruser requires ‘s’ permission for the resource ’0x201f 0xffff 0xab3c34af 0xcdda8cf6 0x91dbfccb 0xdf3fd258′ of class IBM.ResourceGroup on node vmreglnxesx64a.torolab.ibm.com.

(lsrsrc-api) 2610-440 Permission is denied to access a resource specified in this command.\nNetwork Identity regress@0x75bd611f6bab1867/clusteruser requires ‘s’ permission for the resource ’0×2020 0xffff 0xab3c34af 0xcdda8cf6 0x91dbfccd 0x344463a8′ of class IBM.ManagedResource on node vmreglnxesx64a.torolab.ibm.com.

lssam: An unexpected RMC error occurred.The RMC return code was 1.

These messages have no functional impact and can be ignored. This defect will be addressed in SA MP APAR IZ89629.

What to do if I see the following warning in the db2diag.log when running Power 7?

2011-03-17-14.45.22.377107-300 I9247A375          LEVEL: Warning
PID     : 4915288              TID : 1           PROC : db2fmp
INSTANCE: db2inst1             NODE : 000
EDUID   : 1                    EDUNAME: db2fmp
FUNCTION: DB2 UDB, oper system services, sqloJVMvfprintf, probe:20
DATA #1 : String, 63 bytes
Unhandled exception
Type=Segmentation error vmState=0×00000000

You can try running Power 7 in Power 6 compatibility mode or try setting:

JAVA_COMPILER=none

also please confirm that you are on a supported version of JDK and DB2 for Power 7.

Related information

Preparing for DB2 product installation on Power7 System

Oracle Solaris 11 Administrator’s Cheat Sheet

admin — Sun, 01 Apr 2012 23:23:19 +0000

Below is Oracle Solaris 11 Administrator’s Cheat Sheet

Oracle Solaris 11 Cheat Sheet
General Administration
Installation
Automated Installer (AI) is the new network based multi-client provisioning system on
Oracle Solaris 11. AI provides hands-free installation of both SPARC and x86 systems by
using an installation service that installs systems from software package repositories on
the network.
Create an install service from a downloaded ISO file, specifying x86 based DHCP
client starting at address 192.168.1.210 with a total count of 10 addresses:
# installadm create-service -n s11x86 -i 192.168.1.210 -c 10 -s
/path/to/solaris-11-1111-ai-x86.iso
List all enabled services:
# installadm list
List any installation manifests associated with the install services:
# installadm list -m
Export the default installation manifest associated with the s11x86 service:
# installadm export -n s11x86 -m orig_default > manifest.xml
Import a manifest to be associated with the s11x86 service:
# installadm update-manifest -n s11x86 -m orig_default \
-f manifest.xml
List any system configuration profiles associated with the install services:
# installadm list -p
Create a system configuration profile interactively, saving the contents to a file:
# sysconfig create-profile -o profile.xml
Validate a system configuration profile against the default x86 install service:
# installadm validate -n default-i386 -P profile.xml
Associate a system configuration profile with the deafult x86 install service and
give it a name sc-profile:
# installadm create-profile -n default-i386 -f profile.xml \
-p sc-profile
Apply a criteria that all clients must have 4096MB memory or greater to the
manifest s11manifest of s11x86 service:
# installadm set-criteria -m s11manifest -n s11x

Full PDF: Oracle Solaris 11 Administrator’s Cheat Sheet

AIX Support Center Tools web site now available for AIX 7.1, 6.1 and 5.3

admin — Sun, 01 Apr 2012 20:56:43 +0000

The AIX Support Center Tools web site provides information about widely used data gathering tools and recommendation tools that are used by AIX system administrators in conjunction with the IBM support center team. Data gathering tools like zsnap help reduce the amount of time spent during initial problem determination. The recommendation tools help provide health check reports or cross-product compatibility information. Visit AIX Support Center Tools

Featured tools:

snap

Snap captures system configuration information for AIX and PowerHA. Product technical support centers regularly request snap output to identify and resolve problems.

zsnap

The zsnap utility delivers all the benefits of the standard snap command and captures additional debugging data in an easily understood format. IBM’s world-class AIX support centers have teamed up to deliver a more fully integrated tool for data gathering, one that has already been proven to reduce the time needed to resolve customer problems.

perfpmr

The perfpmr tool is used extensively by the AIX technical support centers. This package contains a set of tools and instructions for collecting the data needed to analyze performance problems.

AIX Support Center Tools

AIX Support Center Tools provides information about widely used data gathering tools and recommendation tools that are used by AIX system administrators in conjunction with the IBM support center team. The gathering tools help reduce the amount of time spent during initial problem determination. The recommendation tools help provide health check reports or cross-product compatibility information.

Data gathering tools

snap

The snap command is included with the operating system. Snap captures system configuration information for AIX and PowerHA. Product technical support centers regularly request snap output to identify and resolve problems. Snap output can be delivered to IBM in portable archive exchange format (pax).
zsnap

The zsnap utility delivers all the benefits of the standard snap command and captures additional debugging data in an easily understood format. IBM’s world-class AIX support centers have teamed up to deliver a more fully integrated tool for data gathering, one that has already been proven to reduce the time needed to resolve customer problems. IBM recommends that you install zsnap before you call IBM technical support.
perfpmr

The perfpmr tool is used extensively by the AIX technical support centers. This package contains a set of tools and instructions for collecting the data needed to analyze performance problems. IBM may ask you to download and use this tool.

Recommendation tools

VIOS Advisor

The VIOS Advisor application runs on the PowerVM Virtual I/O Server partition. The application collects key performance metrics before analyzing the results. It then provides a health check report listing recommended changes or areas requiring further investigation.
FLRT

The Fix Level Recommendation Tool web site provides fix recommendations for AIX, PowerHA, VIOS, System Firmware and many key components of your Power Systems server.

How to setup LVM hot spare

admin — Sun, 01 Apr 2012 20:31:23 +0000

Question

Setup LVM hot spare.

Answer

Hot Spare only works if the volume group has mirrored logical volumes. There can be no logical partitions using the drive to be marked a hot spare. The following command marks hdisk1 as a hot spare disk:

# chpv -hy hdisk1

Now that the drive is marked as a hot spare you must determine the hotsparepolicy, and the syncpolicy. Here are your options:

Hotsparepolicy:

y This policy automatically migrates partitions from one failing disk to one
spare disk. From the pool of hot spare disks, the smallest one that is big
enough to substitute for the failing disk will be used.

Y This policy automatically migrates partitions from a failing disk, but might
use the complete pool of hot spare disks.

n No automatic migration will take place. This is the default value for a volume group.

r This value removes all disks from the pool of hot spare disks for this volume group.

Syncpolicy:

y This will automatically try to synchronize stale partitions.

n This will not automatically try to synchronize stale partitions. The latter argument is also the default for a volume group.

This is the syntax to set the policies:

# chvg -h hotsparepolicy -s syncpolicy VolumeGroup

for example:

# chvg -hy -sy vgname

Now the vg will have a hot spare.

AIX Solaris HPUX UNIX Linux system storage administration ksh/perl scripting

Yup, another PowerHA bug! cl_chfs: VG sharedvg is concurrent

Determining if TSAMP/RSCT rebooted a node/server

Problem(Abstract)

Symptom

Resolving the problem

GPFS filesystem outage with a kernel panic 225E3B63

Symptoms

Diagnosis

DBI1122E after using db2iupdt command on existing instance

Problem(Abstract)

Symptom

Cause

Diagnosing the problem

Resolving the problem

Related information

Oracle installation compiler requirement

Compiler Requirements for Oracle Database 11g Release 2(11.2) installation:

Abstract

Download Description

Prerequisites

Installation Instructions

Download package

Technical support

MustGather: No response (hang) on AIX 5.2 and higher versions produces no javacore

Problem(Abstract)

Resolving the problem

SEA Failover flapping on backup SEA

Question

Cause

Answer

Optimizing Sybase Adaptive Server Enterprise (ASE) for IBM AIX

Upgrading CAA in a PowerHA v7.1 Cluster

SAS 9.2 on IBM AIX 7.1: Tuning guideline

SAS on IBM AIX 6.1: Tuning guidelines

SAS AIX 5L Tuning Guide

Running SAS applications on Virtual I/O based storage configuration

Introduction

The SAS test environment

Live Partition Mobility for Oracle RAC

How to Apply ACL to New Files Automatically

Question

Answer

Who’s using my port? (AIX, Linux, Windows example)

Question

Cause

Answer

Setting up your DB2 High Availability (HA) environment on Power7 systems

Abstract

Content

Related information

Oracle Solaris 11 Administrator’s Cheat Sheet

AIX Support Center Tools web site now available for AIX 7.1, 6.1 and 5.3

AIX Support Center Tools

Data gathering tools

snap

zsnap

perfpmr

Recommendation tools

VIOS Advisor

FLRT

How to setup LVM hot spare

Question

Answer