Watch, Follow, &
Connect with Us
Public Report
Report From: Delphi-BCB/Database/TField/TStringField    [ Add a report in this area ]  
Report #:  111942   Status: Closed
[Regression in XE3 Update 1] Data processing performance regression
Project:  Delphi Build #:  17.0.4723.55752
Version:    17.1 Submitted By:   Serge (Developer Express Support)
Report Type:  Basic functionality failure Date Reported:  1/15/2013 3:39:55 AM
Severity:    Serious / Highly visible problem Last Updated: 4/23/2013 7:56:15 AM
Platform:    All versions Internal Tracking #:   35115
Resolution: Fixed (Resolution Comments) Resolved in Build: : XE4
Duplicate of:  None
Voting and Rating
Overall Rating: No Ratings Yet
0.00 out of 5
Total Votes: 68
Description
[Regression in XE3 Update 1]
After installing Update1 for the XE3 IDE, data processing time is significantly increased (by 20 times in certain cases).

We consider this performance issue is very serious regression (see the attached project).
Steps to Reproduce:
1. Run the attached project;
2. Click the "Iterate" button -> time to process data is approximately 4 seconds while before installing Update1 it was about 0.2 seconds.

procedure TForm105.Button1Click(Sender: TObject);
var
  I: Integer;
  V: Variant;
  C: Cardinal;
begin
  C := GetTickCount;
  ClientDataSet1.DisableControls;
  ClientDataSet1.First;
  while not ClientDataSet1.Eof do
  begin
    for I := 0 to ClientDataSet1.FieldCount - 1 do
      V := ClientDataSet1.Fields[I].Value;
    ClientDataSet1.Next;
  end;
  ClientDataSet1.EnableControls;
  Caption := 'Iterate all values = ' + IntToStr(GetTickCount - C) + ' ms';
end;

procedure TForm105.FormCreate(Sender: TObject);
var
  i: Integer;
begin
  ClientDataSet1.DisableControls;
  for i := 1 to 100000 do
  begin
    ClientDataSet1.Append;
    ClientDataSet1bbb.AsString := IntToStr(random(100000));
    ClientDataSet1.Post;
  end;
  ClientDataSet1.EnableControls;
  Caption := 'RecordCount = ' + IntToStr(ClientDataSet1.RecordCount);
end;
Workarounds
None
Attachment
dxSample.zip
Comments

Serge (Developer Express Support) at 1/16/2013 3:23:38 AM -
Hello,

Would you please change the "Area" of this bug report from "Midas\TclientDataSet" to "Database\TField\TStringField" as this issue seems to affect all DataSet TStringField fields?

Tomohiro Takahashi at 1/16/2013 5:14:10 PM -
Thanks for the update. I corrected [Area] filed.

Jan Doggen at 1/17/2013 3:10:54 AM -
I can confirm this.

XE2 executable compiled under XE2.4 measures 94 and 766 ms for 100000 resp 1000000 records.

The same program compiled under XE 3.1 takes 210 and 1910 msecs, i.e. more than double!

Jose Luis Rocha at 3/20/2013 1:10:30 AM -
In XE3, TClientDataset.LoadFromFile, loading a 265.000 records file:

Without MidasLib: 7.86 seconds

With MidasLib: 6 hours and 43 minutes!!!!.

Its a very serious bug, it concerns not only to us as programmers, but also to final applications, to our customers. I need to distribute my app in a single executable, I cannot fix it distributing one more file.

When Embarcadero will fix this bug ?.

We need a fix or a workaround urgently!!!.

Jan Doggen at 1/21/2013 6:11:45 AM -
I have installed 3.0 and then updated to 3.1 and here are new test results:

XE 2.4
100000 iterations    125-  141 msec
1000000 iterations   1188- 1297 msec

XE 3.0
100000 iterations    297-  313 msec  
1000000 iterations   1843- 2875 msec

XE 3.1
100000 iterations   1781- 1851 msec
1000000 iterations  17735-19063 msec

I.e. the update to 3.0 doubles the execution time, and the update to 3.1 multiplies by 6 again(!)

Jan Doggen at 1/24/2013 12:54:12 AM -
It very much looks like the generics are the reason
But now: where is the solution?

Here are my code traces and profiling results

Checking behind the screens what happens with
V := ClientDataSet1.Fields[I].Value;

XE 2 Trace:
===========

{code}
In data.db:

function TFields.GetField(Index: Integer): TField;
begin
  if FSparseFields > 0 then
  begin
    if Index >= FSparseFields then
      TFieldsGetFieldError(Index, DataSet);
    Result := FList[0];
    Result.FOffset := Index;
  end else
    Result := FList[Index];    <==
end;

and then very straightforward into System.Classes:

function TList.Get(Index: Integer): Pointer;
begin
  if Cardinal(Index) >= Cardinal(FCount) then
    Error(@SListIndexError, Index);
  Result := FList[Index];   <==
end;
{code}


XE 3.1 Trace:
============

{code}
In Data.DB:

function TFields.GetField(Index: Integer): TField;
begin
  if FSparseFields > 0 then
  begin
    if Index >= FSparseFields then
      TFieldsGetFieldError(Index, DataSet);
    Result := FList[0];
    Result.FOffset := Index;
  end else
    Result := FList[Index];   <== Leads to:
end;

function TIntegerField.GetAsVariant: Variant;
var
  L: Longint;
begin
  if GetValue(L) then Result := L else Result := Null;
end;

with

function TIntegerField.GetValue(var Value: Longint): Boolean;
var
  Data: TValueBuffer;
begin
  SetLength(Data, SizeOf(Integer));
  Result := GetData(Data);
  case DataType of    <== None of these cases apply in my sample (things would get even worse)
    ftShortint:
      begin
        if Result then
          Value := TBitConverter.ToShortInt(Data);
      end;
    ftByte:
      begin
        if Result then
          Value := TBitConverter.ToByte(Data);
      end;
    ftSmallint:
      begin
        if Result then
          Value := TBitConverter.ToSmallInt(Data);
      end;
    ftWord:
      begin
        if Result then
          Value := TBitConverter.ToWord(Data);
      end;
    ftLongWord:
      begin
        if Result then
          Value := TBitConverter.ToLongWord(Data);
      end;
    else
      begin
        if Result then
          Value := TBitConverter.ToLongInt(Data);
      end;
    end;
end;

and the GetData is:

function TField.GetData(Buffer: TValueBuffer; NativeFormat: Boolean = True): Boolean;
begin
  if FDataSet = nil then DatabaseErrorFmt(SDataSetMissing, [DisplayName]);
  if FValidating then
  begin
    Result := FValueBuffer <> nil;
    if Result and (Buffer <> nil) then
      CopyData(FValueBuffer, Buffer);
  end
  else
    Result := FDataSet.GetFieldData(Self, Buffer, NativeFormat);  <== Leads to:
end;

function TDataSet.GetFieldData(Field: TField; Buffer: TValueBuffer; NativeFormat: Boolean): Boolean;
var
  StackBuf: TArray<Byte>;
begin
  if NativeFormat then
    Result := GetFieldData(Field, Buffer)    
  else
  begin
    if Field.DataSize > dsMaxStringSize + 1 then
      SetLength(StackBuf, Field.DataSize)
    else
      SetLength(StackBuf, dsMaxStringSize + 1);
    Result := GetFieldData(Field, StackBuf);
    if Field.DataType = ftString then
      SetLength(StackBuf, TEncoding.ANSI.GetString(StackBuf).IndexOf(#0));
    if Result then
      DataConvert(Field, StackBuf, Buffer, False);
  end;
end;

with:

function TCustomClientDataSet.GetFieldData(Field: TField; Buffer: TValueBuffer): Boolean;
var
  IsBlank: LongBool;
  RecBuf: PByte;
begin
  Result := False;
  if GetActiveRecBuf(RecBuf) then        <== Returns false
    if Field.FieldKind in [fkData, fkInternalCalc] then  
    begin                                                                                      
      Check(FDSCursor.GetField(RecBuf, Field.FieldNo, Buffer, IsBlank));                        
      Result := not IsBlank;                                                                  
    end                                                                                        
    else if State in [dsBrowse, dsEdit, dsInsert, dsCalcFields] then        
    begin                                                                  
      Inc(RecBuf, FRecordSize + Field.Offset);                            
      Result := Boolean(RecBuf[0]);                                        
      if Result and (Buffer <> nil) then                                                      
        Move(RecBuf[1], Buffer[0], Field.DataSize);                                            
    end;                                                                                      
end;                                                                                          

with:

function TCustomClientDataSet.GetActiveRecBuf(var RecBuf: PByte): Boolean;

  function GetOriginalBuffer: TRecordBuffer;
  begin
    UpdateCursorPos;
    Result := TempBuffer;
    if FDSCursor.GetProp(curpropGETORG_RECBUF, Result) <> DBERR_NONE then
      GetCurrentRecord(Result);
  end;

begin
  case State of
    dsBlockRead,
    dsBrowse: if IsEmpty then
                RecBuf := nil
              else
                RecBuf := @ActiveBuffer[0];       <== Applies
    dsEdit, dsInsert: RecBuf := @ActiveBuffer[0];
    dsSetKey: RecBuf := PByte(FKeyBuffer) + SizeOf(TKeyBuffer);
    dsCalcFields,
    dsInternalCalc: RecBuf := CalcBuffer;
    dsFilter: RecBuf := FFilterBuffer;
    dsNewValue: RecBuf := @FNewValueBuffer[0];
    dsOldValue: if FOldValueBuffer <> nil then
                  RecBuf := @FOldValueBuffer[0]
                else
                  RecBuf := @GetOriginalBuffer[0];
    dsCurValue: RecBuf := @FCurValueBuffer[0];
    dsInActive: RecBuf := nil;
  else
    RecBuf := nil;
  end;
  Result := RecBuf <> nil;  <== false
end;
{code}


I have used AQTime to do some profiling and it shows these horrible results:

{code}
Performance hot spots

Worst performance (body only):

TStringField::GetValue  1,85  
TCustomClientDataSet::InternalPost  0,37  
TCustomClientDataSet::GetRecord  0,17  
TDataSet::DataEvent  0,13  
TCustomClientDataSet::DataEvent  0,10  
TFields::GetField  0,08  
TDataSet::CalculateFields  0,07  
TCustomClientDataSet::SetFieldData  0,07  
TCustomClientDataSet::Check  0,07  
TCustomClientDataSet::SaveDataPacket  0,06  

Worst performance (with children):

TFrmXE3ClientDatasetSpeed::Test  2,50  
TStringField::GetAsVariant  1,96  
TStringField::GetValue  1,94  
TCustomClientDataSet::Post  1,02  
TDataSet::Post  0,99  
TDataSet::CheckOperation  0,50  
TCustomClientDataSet::InternalPost  0,43  
TCustomClientDataSet::DataEvent  0,42  
TDataSet::Append  0,39  
TDataSet::Next  0,38  

Routine with max HitCount:

System::Generics::Collections::TList__1<Data::Db::TField *>::GetItem  1200037  
TFields::GetField  1200036  
TDataSet::ControlsDisabled  1200012  
TCustomClientDataSet::DataEvent  1000010  
TDataSet::DataEvent  1000010  
TCustomClientDataSet::Check  900008  
TFields::GetCount  900008  
TField::GetFieldNo  400012  
TCustomClientDataSet::ClearCalcFields  400000  
TCustomClientDataSet::SetAltRecBuffers  300005  
{code}

Jan Doggen at 1/28/2013 2:09:02 AM -
No changes in XE3 Update 2 (but then, it wasn't in the fix list)

Robert Triest at 2/5/2013 12:51:46 AM -
We have two versions of XE3 waiting to be used but there are/were too much issues on database level. (BDE support, DataSnap, this report..) Please make sure that at least all database related support of Delphi are working 100% because this is the main use of Delphi for the most developers!!

Jan Doggen at 2/12/2013 1:32:46 AM -
I notice that this issue is now marked as 'Resolved' ('checked in at 2/8/2013 4:55:40 PM') but I see no resolution to the isse. as reported earlier, XE3 update 2 did not fix this.

craig oberfield at 4/1/2013 5:49:31 AM -
Where is the resolution?  Was this marked "Resolved" by accident?  Delphi should be getting faster not slower or at least staying the same.  Speed is Delphi's greatest asset and I'm sure they take this issue very seriously.

Please post the fix / workaround steps.

Tomohiro Takahashi at 2/12/2013 5:41:19 AM -
This issue is not closed(w/ fixed) yet.

Russell Weetch at 4/15/2013 7:25:57 AM -
Can we please have an update on this as I have a load of work going to waste at the moment

Russell Weetch at 3/26/2013 5:24:06 AM -
When can we expect this fix to be shipped?

Oliver Rick at 4/2/2013 11:21:20 AM -
I would also really need this bug to be fixed. This bug stopped my whole developement in XE3, because i can't deliver slower applications than before to my clients.

Poul Halgaard at 4/5/2013 1:55:02 AM -
I was the one causing developer express to report this in quality central originally. I thought it was their components causing this and it turned out it was Delphi. It took quite some hours to investigate. DevExpress was very helpfull.
I actually released a version build in XE3 and took a lot of beating from my customers because of the new (slow) speed. I had to quicky step back to XE2 and release again. This was not a nice experience!

XE3 is completely useless to me thanks to this!! Seeing this is not yet fixed frustrates me a lot. Why don't Embarcadero comment?!

Solutions EUM at 4/26/2013 12:09:28 AM -
When updated 3 ?

Solutions EUM at 5/9/2013 10:07:45 PM -
Why do not the answer?

Tomohiro Takahashi at 5/10/2013 12:22:46 AM -
If possible, please contact technical support or Sales rep of Embarcadero. Sorry for the inconvenience.
http://support.embarcadero.com/

Solutions EUM at 5/15/2013 5:07:25 PM -
Embarcadero should have fix it  in a hot-fix in XE3, not only in XE4.

Jose Luis Rocha at 4/26/2013 1:07:23 AM -
Fixed too late. Embarcadero should have fixed it urgently in a hot-fix in XE3, not only in XE4.

Jon Stefansson at 11/27/2013 7:48:36 AM -
I agree. This is extremely inconvenient. It also makes you ask why buy XE4 to get a fix for an admitted error, when the solution to the next error in XE4 will be to buy XE5. That's a vicious circle..

Server Response from: ETNACODE01