[#QA-54] Unable to close database by srvctl and racgimon takes 100% of CPU.

Questions & Answers

Unable to close database by srvctl and racgimon takes 100% of CPU.

Created: 26/Mar/13 02:47 PM Updated: 26/Mar/13 03:22 PM

Return to search

Issue 4 of 9 issue(s)

<< Previous | QA-54 | Next >>

Fix Version/s:

None

Product Version:	11.2.0.3
Operating System:	Solaris
Host Name:	.
Database Name:	.

Description

« Hide

Unable to close the database:

$ srvctl stop database -d ESIBASE
PRKP-1002 : Error stopping instance ESIBASE1 on node ersteracsrv1
CRS-0216: Could not stop resource 'ora.ESIBASE.ESIBASE1.inst'.
$

2 racgimon processes take 100% of CPU in prstat output:

  PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
  8286 oracle    64  36 0.0 0.0 0.0 0.0 0.0 0.0   0  37 .15   0 racgimon/1
  7903 oracle    65  35 0.0 0.0 0.0 0.0 0.0 0.0   0  35 .15   0 racgimon/1
 10015 root     0.0 0.8 0.0 0.0 0.0 0.0  99 0.0  21   1 398   0 prstat/1
  7818 oracle   0.2 0.0 0.0 0.0 0.0 0.0 100 0.0  62   1  7K   0 oracle/2
 10055 root     0.0 0.1 0.0 0.0 0.0 0.0 100 0.0   7   0 318   0 sleep/1
   816 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0  30   1 275   0 init.cssd/1
  1916 oracle   0.1 0.0 0.0 0.0 0.0 0.0 100 0.0  60   0 719  59 oracle/1
  1878 oracle   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 170   0 694   1 oracle/1
  1874 oracle   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0 170   0 691   1 oracle/1
  1872 oracle   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0  81   0 401  31 oracle/1
  1621 oracle   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0  53   0 343   1 oracle/1
  1894 oracle   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0  21   0  54   0 oracle/2
  1625 oracle   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0  64   0 201   1 oracle/1
  1623 oracle   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0  64   0 201   1 oracle/1
  1870 oracle   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0  59   0 347   1 oracle/2
 NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU
    72 oracle     21G   21G    57%   0:07:37  26%
    54 root      117M  180M   0.5%   0:00:06 0.2%
     1 noaccess  136M  207M   0.6%   0:00:12 0.0%
     6 daemon   6408K 7496K   0.0%   0:00:00 0.0%
     1 smmsp    1136K 7244K   0.0%   0:00:00 0.0%
Total: 134 processes, 439 lwps, load averages: 2.26, 1.82, 1.13
#

All

Comments

Change History

Sort Order:

Ascending order - Click to sort in descending order

[ Permlink | « Hide ]

ubTools Support - 26/Mar/13 03:01 PM

ANALYSIS 1:

truss output of one of _racgimon:

# truss -fae -p 8286

8286:   close(346745079)                                Err#9 EBADF
8286:   close(346745080)                                Err#9 EBADF
8286:   close(346745081)                                Err#9 EBADF
8286:   close(346745082)                                Err#9 EBADF
8286:   close(346745083)                                Err#9 EBADF
8286:   close(346745084)                                Err#9 EBADF
8286:   close(346745085)                                Err#9 EBADF
8286:   close(346745086)                                Err#9 EBADF

# truss -faec -p 8286
psargs: /u01/app/oracle/product/10.2/bin/racgimon startd ESIBASE
^C
syscall               seconds   calls  errors
close                   2.374 1857265 1857265
                     --------  ------   ----
sys totals:             2.374 1857265 1857265
usr time:               1.079
elapsed:               23.090
#

Comment:

racgimon could not close file descriptors. It repeats to close different file descriptors which are incremented 1 in each subsequent close() system call.

close() system calls return EBADF, which is The fildes argument is not a valid file descriptor.
Ref: http://docs.oracle.com/cd/E23823_01/html/816-5167/close-2.html#REFMAN2close-2

[ Permlink | « Hide ]

ubTools Support - 26/Mar/13 03:09 PM

ANALYSIS 2:

prctl outpur of racgimon:

# prctl 8286
process: 8286: /u01/app/oracle/product/10.2/bin/racgimon startd ESIBASE
NAME    PRIVILEGE       VALUE    FLAG   ACTION
RECIPIENT
process.max-port-events
        privileged      65.5K       -   deny
 -
        system          2.15G     max   deny
 -
process.max-msg-messages
        privileged      8.19K       -   deny
 -
        system          4.29G     max   deny
 -
process.max-msg-qbytes
        privileged      64.0KB      -   deny
 -
        system          16.0EB    max   deny
 -
process.max-sem-ops
        privileged        512       -   deny
 -
        system          2.15G     max   deny
 -
process.max-sem-nsems
        privileged        512       -   deny
 -
        system          32.8K     max   deny
 -
process.max-address-space
        privileged      16.0EB    max   deny
 -
        system          16.0EB    max   deny
 -
process.max-file-descriptor
        privileged      2.15G     max   deny
 -
        system          2.15G     max   deny
 -
process.max-core-size
        basic               0B      -   deny
8286
        system          8.00EB    max   deny
 -
process.max-stack-size
        basic           10.0MB      -   deny
8286
        privileged       125TB      -   deny
 -
        system           125TB    max   deny
 -
process.max-data-size
        privileged      16.0EB    max   deny
 -
        system          16.0EB    max   deny
 -
process.max-file-size
        privileged      8.00EB    max   deny,signal=XFSZ
 -
        system          8.00EB    max   deny
 -
process.max-cpu-time
        privileged      18.4Es    inf   signal=XCPU
 -
        system          18.4Es    inf   none
 -
task.max-cpu-time
        system          18.4Es    inf   none
 -
task.max-lwps
        system          2.15G     max   deny
 -
project.max-contracts
        privileged      10.0K       -   deny
 -
        system          2.15G     max   deny
 -
project.max-device-locked-memory
        privileged      2.19GB      -   deny
 -
        system          16.0EB    max   deny
 -
project.max-locked-memory
        system          16.0EB    max   deny
 -
project.max-port-ids
        privileged      8.19K       -   deny
 -
        system          65.5K     max   deny
 -
project.max-shm-memory
        privileged      24.0GB      -   deny
 -
        system          16.0EB    max   deny
 -
project.max-shm-ids
        privileged        128       -   deny
 -
        system          16.8M     max   deny
 -
project.max-msg-ids
        privileged        128       -   deny
 -
        system          16.8M     max   deny
 -
project.max-sem-ids
        privileged        128       -   deny
 -
        system          16.8M     max   deny
 -
project.max-crypto-memory
        privileged      8.77GB      -   deny
 -
        system          16.0EB    max   deny
 -
project.max-tasks
        system          2.15G     max   deny
 -
project.max-lwps
        system          2.15G     max   deny
 -
project.cpu-cap
        system          4.29G     inf   deny
 -
project.cpu-shares
        privileged          1       -   none
 -
        system          65.5K     max   none
 -
zone.max-swap
        system          16.0EB    max   deny
 -
zone.max-locked-memory
        system          16.0EB    max   deny
 -
zone.max-shm-memory
        system          16.0EB    max   deny
 -
zone.max-shm-ids
        system          16.8M     max   deny
 -
zone.max-sem-ids
        system          16.8M     max   deny
 -
zone.max-msg-ids
        system          16.8M     max   deny
 -
zone.max-lwps
        system          2.15G     max   deny
 -
zone.cpu-cap
        system          4.29G     inf   deny
 -
zone.cpu-shares
        privileged          1       -   none
 -
        system          65.5K     max   none
 -

$ prctl -n process.max-file-descriptor -i process $$
process: 7615: -sh
NAME    PRIVILEGE       VALUE    FLAG   ACTION
RECIPIENT
process.max-file-descriptor
        basic           4.10K       -   deny
7615
        system          2.15G     max   deny
 -
$

Comment:

privileged option of process.max-file-descriptor had reached to 2.15G descriptors. But, no privileged option had been set to it.

[ Show » ]

ubTools Support - 26/Mar/13 03:09 PM ANALYSIS 2: prctl outpur of racgimon:

# prctl 8286
process: 8286: /u01/app/oracle/product/10.2/bin/racgimon startd ESIBASE
NAME    PRIVILEGE       VALUE    FLAG   ACTION
RECIPIENT
process.max-port-events
        privileged      65.5K       -   deny
 -
        system          2.15G     max   deny
 -
process.max-msg-messages
        privileged      8.19K       -   deny
 -
        system          4.29G     max   deny
 -
process.max-msg-qbytes
        privileged      64.0KB      -   deny
 -
        system          16.0EB    max   deny
 -
process.max-sem-ops
        privileged        512       -   deny
 -
        system          2.15G     max   deny
 -
process.max-sem-nsems
        privileged        512       -   deny
 -
        system          32.8K     max   deny
 -
process.max-address-space
        privileged      16.0EB    max   deny
 -
        system          16.0EB    max   deny
 -
process.max-file-descriptor
        privileged      2.15G     max   deny
 -
        system          2.15G     max   deny
 -
process.max-core-size
        basic               0B      -   deny
8286
        system          8.00EB    max   deny
 -
process.max-stack-size
        basic           10.0MB      -   deny
8286
        privileged       125TB      -   deny
 -
        system           125TB    max   deny
 -
process.max-data-size
        privileged      16.0EB    max   deny
 -
        system          16.0EB    max   deny
 -
process.max-file-size
        privileged      8.00EB    max   deny,signal=XFSZ
 -
        system          8.00EB    max   deny
 -
process.max-cpu-time
        privileged      18.4Es    inf   signal=XCPU
 -
        system          18.4Es    inf   none
 -
task.max-cpu-time
        system          18.4Es    inf   none
 -
task.max-lwps
        system          2.15G     max   deny
 -
project.max-contracts
        privileged      10.0K       -   deny
 -
        system          2.15G     max   deny
 -
project.max-device-locked-memory
        privileged      2.19GB      -   deny
 -
        system          16.0EB    max   deny
 -
project.max-locked-memory
        system          16.0EB    max   deny
 -
project.max-port-ids
        privileged      8.19K       -   deny
 -
        system          65.5K     max   deny
 -
project.max-shm-memory
        privileged      24.0GB      -   deny
 -
        system          16.0EB    max   deny
 -
project.max-shm-ids
        privileged        128       -   deny
 -
        system          16.8M     max   deny
 -
project.max-msg-ids
        privileged        128       -   deny
 -
        system          16.8M     max   deny
 -
project.max-sem-ids
        privileged        128       -   deny
 -
        system          16.8M     max   deny
 -
project.max-crypto-memory
        privileged      8.77GB      -   deny
 -
        system          16.0EB    max   deny
 -
project.max-tasks
        system          2.15G     max   deny
 -
project.max-lwps
        system          2.15G     max   deny
 -
project.cpu-cap
        system          4.29G     inf   deny
 -
project.cpu-shares
        privileged          1       -   none
 -
        system          65.5K     max   none
 -
zone.max-swap
        system          16.0EB    max   deny
 -
zone.max-locked-memory
        system          16.0EB    max   deny
 -
zone.max-shm-memory
        system          16.0EB    max   deny
 -
zone.max-shm-ids
        system          16.8M     max   deny
 -
zone.max-sem-ids
        system          16.8M     max   deny
 -
zone.max-msg-ids
        system          16.8M     max   deny
 -
zone.max-lwps
        system          2.15G     max   deny
 -
zone.cpu-cap
        system          4.29G     inf   deny
 -
zone.cpu-shares
        privileged          1       -   none
 -
        system          65.5K     max   none
 -

$ prctl -n process.max-file-descriptor -i process $$
process: 7615: -sh
NAME    PRIVILEGE       VALUE    FLAG   ACTION
RECIPIENT
process.max-file-descriptor
        basic           4.10K       -   deny
7615
        system          2.15G     max   deny
 -
$

Comment: privileged option of process.max-file-descriptor had reached to 2.15G descriptors. But, no privileged option had been set to it.

[ Permlink | « Hide ]

ubTools Support - 26/Mar/13 03:21 PM

WORKAROUND:

Set privileged option to a value as an example below:

# projmod -s -K "process.max-file-descriptor=(basic,4096,deny),(privileged,65536,deny)" 'user.oracle'

After setting, check as below:

$ prctl -n process.max-file-descriptor -i process $$
process: 708: -sh
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
process.max-file-descriptor
        basic           4.10K       -   deny                               708
        privileged      65.5K       -   deny                                 -
        system          2.15G     max   deny                                 -
$

See similar problem for lower Oracle versions in Oracle note srvctl Slow or Fails to Start/Stop Database Instance and crsd.bin/racgmain/racgimon High CPU Usage [ID 1457387.1].