Watch, Follow, &
Connect with Us
Public Report
Report From: InterBase/Server    [ Add a report in this area ]  
Report #:  120855   Status: Closed
Server crash with maybe some usefull info in log for developers
Project:  InterBase Build #:  WI-V10.0.5.595
Version:    10.0 Submitted By:   Karol Bieniaszewski
Report Type:  Crash / Data loss / Total failure Date Reported:  11/29/2013 5:21:05 AM
Severity:    Critical / Show Stopper Last Updated: 5/8/2015 9:24:01 AM
Platform:    Windows 95/98/NT 4.0 on Intel Internal Tracking #:  
Resolution: Fixed (Resolution Comments) Resolved in Build: : 12.0.2.299
Duplicate of:  None
Voting and Rating
Overall Rating: No Ratings Yet
0.00 out of 5
Total Votes: None
Description
Server crash with maybe some usefull info in log
If you have some light on this


Platform
Windows Server 2008 R2 Enterprise
Intel Xeon E5440
12GB RAM
64 bit

INTERBASE1 (Server)
Fri Nov 29 13:53:09 2013

process_packet() servicing opcode=21
   Access violation.


The code attempted to access a virtual


address without privilege to do so.

This exception will cause the InterBase server

to terminate abnormally.

INTERBASE1 (Server)
Fri Nov 29 13:53:12 2013

process_packet() servicing opcode=63
   Access violation.


The code attempted to access a virtual


address without privilege to do so.

This exception will cause the InterBase server

to terminate abnormally.

INTERBASE1 (Server)
Fri Nov 29 13:53:12 2013

process_packet() servicing opcode=65
   Access violation.


The code attempted to access a virtual


address without privilege to do so.

This exception will cause the InterBase server

to terminate abnormally.

INTERBASE1 (Server)
Fri Nov 29 13:53:13 2013

gds__free: attempt to release bad block


INTERBASE1 (Server)
Fri Nov 29 13:53:14 2013

gds__free: attempt to release bad block


INTERBASE1 (Server)
Fri Nov 29 13:53:14 2013

gds__kb_alloc: memory pool corrupted


INTERBASE1 (Server)
Fri Nov 29 13:53:14 2013

process_packet() servicing opcode=68
   Access violation.


The code attempted to access a virtual


address without privilege to do so.

This exception will cause the InterBase server

to terminate abnormally.

INTERBASE1 (Client)
Fri Nov 29 13:53:14 2013

C:\InterBase\bin\ibserver.exe: terminated abnormally (-1)


INTERBASE1 (Server)
Fri Nov 29 13:53:16 2013

Server: version 10.0.5.595 (64-bit), ibconfig settings...
        SWEEP_QUANTUM to 250, USER_QUANTUM to 1000,
        SWEEP_YIELD_TIME to 1 ms, and MAX_THREADS to 1000000
        SQL_COMPILER_RECURSION to 2000


INTERBASE1 (Client)
Fri Nov 29 13:53:17 2013

Guardian starting: C:\InterBase\bin\ibserver.exe
Steps to Reproduce:
None
Workarounds
None
Attachment
LogAndConfig.zip
Comments

Karol Bieniaszewski at 2/4/2014 11:57:02 PM -
>>Do you have any other useful messages in interbase.log?

only logs like the following (first line is repeated many times) - but this are i suppose different situation..

INTERBASE1 (Server)
Fri Jan 31 06:03:29 2014

INET/inet_error: send errno = 10053 client host = INTERBASE1 connection name = INTERBASE1 user name = SYSTEM server port number = 0

INTERBASE1 (Client)
Fri Jan 31 06:03:29 2014

INET/inet_error: read errno = 10054 client host = UNKNOWN connection name = 10.101.200.45 user name = UNKNOWN server port number = 3050

INTERBASE1 (Client)
Fri Jan 31 06:03:29 2014

REMOTE INTERFACE/gds__detach: Unsuccessful detach from database.

Uncommitted work may have been lost

INTERBASE1 (Server)
Fri Jan 31 06:03:30 2014

Server: version 10.0.5.595 (64-bit), ibconfig settings...
        SWEEP_QUANTUM to 250, USER_QUANTUM to 1000,
        SWEEP_YIELD_TIME to 1 ms, and MAX_THREADS to 1000000
        SQL_COMPILER_RECURSION to 2000


INTERBASE1 (Client)
Fri Jan 31 06:03:30 2014

Guardian starting: C:\InterBase\bin\ibserver.exe

Karol Bieniaszewski at 2/5/2014 12:26:00 AM -
I add ibconfig to attachements - may be there is some interesting modification

In near future we will upgrade to XE3 then we see

Sriram Balasubramanian at 2/6/2014 10:44:20 AM -
Karol,
The only problem I see with your ibconfig file is the TMP_DIRECTORY settings. They need to have a size specified. Here's the new modified instructions for the parameter (it was wrongly worded in the previous releases).

#TMP_DIRECTORY
## Platforms: All
## Directory to use for storing temporary files.
## Specify directory path and number of bytes available in the directory.
## Directory path MUST be enclosed in double quotes.
## Format for entry is as follows.
##     TMP_DIRECTORY <MaxBytes> <AbsoluteDirectoryPath>
##     MaxBytes should be a number indicating maximum space in bytes
##       to be used in the directory, of type 32-bit unsigned integer.
##       Valid range: 1MB - 4GB (in bytes)
## For e.g.
##     TMP_DIRECTORY 500400300 "C:\Temp"
##     TMP_DIRECTORY 1500400300 "/tmpdir"
## List multiple entries, one per line; directories are
## used in the order specified.
## Default is the value of the INTERBASE_TMP environment
## variable; otherwise /tmp on UNIX or C:\temp on Windows NT/2000/XP

Karol Bieniaszewski at 2/6/2014 11:35:07 PM -
Is possible to set temp without limitation? I do not need to limit temp dir to e.g. 4GB i have 200GB free for temp..

Sriram Balasubramanian at 2/7/2014 9:47:38 AM -
Karol,

Since you have a lot of temp space, you can comment out your TMP_DIRECTORY settings in ibconfig and just define your INTERBASE_TMP or TMP environment variables and restart the server.

Documented in OpGuide.pdf (http://docs.embarcadero.com/products/interbase/IBXE3Update3/OpGuide.pdf), Chapter 4, section "Managing Temporary Files"

If you specify temporary directories in ibconfig, the server uses those values for
the sort files and ignores the server environment variable values. If you don’t
specify configuration of temporary directories in ibconfig, then the server picks a
location for a sort file based on the following algorithm:4-20 Operations Guide
Configuring Parameters in ibconfig
1 Use the directory defined in INTERBASE_TMP environment variable (in server space, or, system wide)
2 If INTERBASE_TMP isn’t defined, use directory defined in TMP environment variable
3 If TMP isn’t defined, default to the /tmp directory (UNIX) or C:\temp (Windows)

Karol Bieniaszewski at 2/9/2014 2:39:36 AM -
Now this is clear, Thanks

Will be good to see this comment in ibconfig comment section :)

Sriram Balasubramanian at 2/10/2014 10:57:51 AM -
Great; additional comments to the default ibconfig file have been made and checked in for a future update.

Karol Bieniaszewski at 2/6/2014 11:23:08 PM -
Ok i try to fix config

But i test now our task on XE3 and we also have crash and we know now what couse the crash

i attached log from Interbase

i try to explain task details
We have 16 databases and we consolidate them to one database in night work
task have steeps like this
1. Start transaction
2. delete from table "details1" ~734832 rows
3. delete from table "details2" ~500000 rows
4. delete from table "header" 717195
5. copy data from external tables
6. commit

on table details and details2 and header are triggers with calculate aggregation fields for table header

it looks on details1 and details2 like
update header key=key where key = key_from_detail

you know we change nothing but update action activate trigger on header table
where is something like this not exactly but concept you see

SELECT SUM(VALUE) FROM DETAIL1 INTO HEDER_FIELD_SUM1;
SELECT SUM(VALUE) FROM DETAIL2 INTO HEDER_FIELD_SUM2;


and all work ok without termination server - but on production server we havae monitoring software run in intervale 5 minutes wchich do something like this
1. try to connect to database - raport ok or not
2. SELECT 10 records from some table and see if we got this 10 records - raport ok or not
3. SELECT * FROM TMP$DATABASE - raport MEM STATS

and we see that server not crash if we remove steep 3 from monitoring software
Selecting from TMP$DATABASE couse server to crash

Sriram Balasubramanian at 2/7/2014 9:34:57 AM -
Karol,

Interesting. Glad you were able to isolate the crash to the query on TMP$DATABASE. When executing a query on any Performance Monitoring table (such as TMP$DATABASE), the whole system/server is put "on hold" for other running requests while the PM data is refreshed. It seems like one of the requests is not coming out well after coming out of the "hold" state.

Since you have now reproduced the problem with InterBase XE3, I suggest you build us a reproducible test case (with your database) and submit it to Embarcadero Support. We can then use it reproduce the problem for resolution. Thanks!

Based on what you have just found with your production server, I assume you have now implemented a workaround to disable the query on the PM table TMP$DATABASE.

Karol Bieniaszewski at 2/9/2014 2:42:24 AM -
Yes i disable any TMP$ tables usage.

I can not send our db and project
and do not know if i'am able to create reproductible test case :/

Sriram Balasubramanian at 2/10/2014 10:57:07 AM -
Karol,

Ok; let us know if and when you can. In the meantime, we will try to reproduce the problem on our end as well. Thanks!

Karol Bieniaszewski at 5/7/2015 12:42:15 AM -
I suppose that this ticket can be closed as Fixed in XE7 Update 2
look at my raports in support tickets INTB-2391, INTB-2420

Server Response from: ETNACODE01