DelphiTools.info

Very experimental support for LLVM in DWScript is now in the googlecode repository, thanks to the efforts of Christian-W Budde.

This is the beginning of the opening of new realms of possibilities for DWS!

LLVM support is currently centered around jit compilation, and at the moment doesn’t extend much beyond the needs of the Mandelbrot demo (see below for what is supported right now).

LLVM CodeGen vs dwsJIT

Note that it is a different effort from the recently announced JITter for DWS, the TdwsJIT is a simple greedy just in time compiler, which works on the same structures as the interpreter, and where a goal is that jitting should be lightweight and fast enough to be applied all the time. It’s a “cheap” boost to script execution, and a hybrid architecture (cf. future article).

The LLVM codegen on the other hand involves a bit more overhead, in terms of a large DLL and longer compilation/JITting times, but has a wealth of other benefits.

The LLVM codegen isn’t tied to the interpreted structures, so it means more work down the road to get everything working, but there is much greater optimization potential, and the natively compiled code can be cached or persisted (thus mitigating or negating the compilation overhead).

Finally LLVM is of course cross-platform, and there is the potential for compiling standalone binaries as well…

Performance

No exact performance figures just yet, it’s too early and they change too often!

From initial tests, LLVM with optimization level zero (no optimization) is in the same ballpark as the dwsJIT, though with a slight JIT overhead (dwsJIT has none).

When increasing the optimization level, LLVM takes a measurable amount of time, but produces the best compiled binary on the benchmark so far! It’s only behind the hand-optimized version, and not all optimizations in LLVM are active just yet…

What’s working

Note that this is a very temporary list, as things change rapidly, so use it more as “cheat list” for what you can test right now.

What is working so far:

• Primitive expressions
  • Integer
    • Arithmetrical (Add, Sub, Mul, Div, Mod, Neg, Sqr & Abs)
    • Arithmetrical assigns (+=, *=, -=, …, see above)
    • Bit Manipulations (Sar, Shr, Shl, And, Or, Xor, Not)
    • Comparisons (Less, LessEqual, Equal, GreaterEqual, Greater)
    • Shortcuts such as Inc & Dec
    • Min & Max (inlined)
  • Float
    • Arithmetrical (Add, Sub, Mul, Div, Neg, Sqr, Abs, Sqrt, Sin, Cos, Power, Exp, Ln, Log10, Log2)
    • Arithmetrical assigns (+=, *=, -=, …, see above)
    • Conversions (Floor, Ceil, Trunc, Round)
    • Comparisons (Less, LessEqual, Equal, GreaterEqual, Greater)
    • Min & Max (inlined)
  • Boolean
    • Logical (And, Or, Xor, Not)
• Aggregate Types
  • Arrays
    • simple array types (read only!)
    • only primitive types
  • Records
    • simple record types
    • nesting is not thoroughly tested yet
    • may not contain arrays
  • Strings
    • only constant strings
• Control Flow Expressions
  • Branches
    • If / Then
    • If / Then / Else
    • If / Then / Else -> Value
  • Loops
    • while
    • repeat
    • for (upward / downward, fixed / variable steps)
  • Functions
    • using simple variables (no var or const yet)
    • main function (typically without declaration in DWS)
    • may be called from other functions
    • PrintLn (to print constant strings, rely on external definition)

And here’s the list of things that are not working:

• Strings and string management (except for string constants)
  • obviously no helpers as well
• Resource strings
• Dynamic arrays
• Variants
• complex array types (containing non-primitive values)
• complex record types (containing non-primitive values)
• Classes
  • Fields
  • Methods
  • Constructors
  • Self
  • Visibility
• Interfaces
• Objects
• Case
• Special loop aware
  • ForCharIn / ForCharCodeIn
  • Continue / Break / Exit / Exit(Value)
• Exceptions (try, finally, except, raise, assert)
• Helpers
• Connectors
• Special math functions like Tan, Arc*, etc.

Eric News DWS, JIT, LLVM

Language and Script Engine

• Delphi XE4 is now supported, compiler hints for XE3 have been taken care of as well
• compiler now supports “in” operator across strings, for for instance “if subString in myString then” is equivalent to “if myString.Contains(subString) then“
• added standard “helpers” for most built-in functions operating on String, Integer, Float and Boolean. These roughly follow the Delphi & .Net conventions, though standard Maths functions are accessible directly (f.i. you can do “angle.Cos” directly, rather than have to go through “Math.Cos(angle)“)
• added some more string functions (DeleteLeft, DeleteRight, etc.)
• added support for EmptyParam for the COM connector
• added DivMod to the standard functions
• improved performance of DecodeDate/Time functions
• improvements and fixes for the JSON connector (now supports manipulating an existing JSON structure, creating, moving and deleting nodes)
• misc. fixes and improvements

Note that there aren’t specifics DPKs for XE4, you can just copy/rename the XE2 or XE DPKs.

Changes for SmartMobileStudio / JavaScript Codegen

In addition to the  previous changes, the following will apply to the next version of SmartMS:

• support “for str in variant” which allows enumerating members of a raw JavaScript object (compiles to “for (str in v)”)
• improved code generation for functions with an “exit” statement
• added specific optimizations for string copies and testing (startswidth/endwidth)
• slightly faster code generation

 

Eric News DWS, OP4JS

The second Smart Context 2013 has been announced! As with the previous round, first prize is a tablet device of your own choice (up to USD 750). This time the theme is “Game Development”

The rules are as follows:

• Registration before the 13th of May (registration at contest@smartmobilestudio.com)
• Deliver your contribution before 3rd of June
• The source code will be shared in our show case area
• Preferable that it can run in the integrated IDE browser (but not mandatory)
• No restrictions w.r.t game genre
• No restrictions w.r.t project type (canvas, sprite, console, VCL)
• You can use the trial version to make your entry

The registration date limit is not intended to be binding, but more like an “Hey I’m in”, and for the previous contest latecomers were accepted as well.

For this round I’ll be one of the judges alongside Jon Lennart Aasenden, who recently posted two articles about game development, see part1 and part2.

For more details, head to the announcement page.

Eric News OP4JS, SmartMS

This is a practical poll question, what would get you to buy a newer Delphi version?
What would you like to see most and foremost, and would most have a use for?
To force you to choose, you can only pick two items!

Eric News Delphi

For a few weeks now, an experimental JIT compiler has been available in the DWScript SVN for 32bits code. A more detailed article into the hows and whats and WTFs will come later.

Consider this article as an extended teaser, and a sort of call for test cases, benchmarks and eyeballs on the source code.

Following benchmarks were compiled in XE2 with full optimization, no stack frames and no range checking. Absolute values aren’t meaningful, just consider them relatively.

Mandelbrot fractal

(lower is better)

Delphi XE2-32bit : 515
Delphi XE3-64bit: 162
DWScript JIT 32bit: 281

The JIT was initially tested on the Mandelbrot benchmark, so that’s one of the cases where JITing is almost complete, with the exception of the SetPixel() call.

SciMark 2

(higher is better)

Delphi XE2-32bit : 507
Delphi XE3-64bit: 682
DWScript JIT 32bit: 215

Delphi version uses pointers, DWScript version was slightly updated to use dynamic arrays instead  and JITting is partial at the moment.

The benchmark involves fairly large matrices, and DWScript use of Variant (16 bytes) rather than Double (8 bytes) means the data no longer fits in the CPU cache, which partly accounts for the poor showing of the JIT.

Array statistics

(lower is better)

Delphi XE2-32bit : 208
Delphi XE3-64bit: 47
DWScript JIT 32bit: 63

This test measures of execution time of the following code (fully JITted), which computes the base values for common array statistics (range, average, deviation, etc.). The Delphi 32bit compiler really suffers because of Min/Max (despite having inlined them).

// "a" a floating point array of non-ordered values
for v in a do begin
   s := s + v;
   s2 := s2 + v*v;
   mi := Min(mi, v);
   ma := Max(ma, v);
end;

What next?

The DWScript JIT compiler relies on SSE2 to outperform the Delphi 32bit compiler, its current main limitations are:

• JIT centers around floating point and a limited subset of integer and Boolean operations, the rest isn’t JITted yet.
• Function calls aren’t JITted at the moment, and neither are a variety of other statements.
• The JIT works with the same data structures as the interpreted engine, that means script debuggers and everything else works on JITted code as if it was still interpreted, but that also means the basic data unit is still the 16 bytes Variant at the moment.
• The JIT register allocator is currently limited to floating point (ie. no integer or pointer allocations).
• DWScript Integer type is 64bit sized, so for 32bit values, Integer performance is lower than what Delphi 32 can do, even though the JIT can generate typically faster code for it than the Delphi 32bit compiler does for Int64.

The JIT also suffers against a 64bit compiler as there are 64bit CPU instructions (and registers) not accessible in 32bit mode, but a 64bit JIT should be able to go farther.

If you’re interested and want to help, I’m currently looking for benchmarks and test cases, if you have code that compiles in both Delphi and DWScript,  particularly on integer maths (encryption, etc.) or object-oriented manipulations (graphs, trees…) that could help. You’re even allowed to have the Delphi version use pointers and other tricks, the comparison doesn’t need to be fair

Eric News Delphi, DWS, JIT

Answering the call for showcases, Steffen Binas prepared a very nice set of screenshots for AquaSoft‘s Slide Show impressive product, which went far beyond the minimal requirement, so I’m posting a full-blown article about it. Following is the presentation of their software and how DWScript is used inside it.

When you need a powerful showcase I’d bet AquaSoft SlideShow is the one: http://www.aquasoft.de/diashow.as (an english website for version 8 is coming soon).

Despite the simple name of the application it’s very powerful, just have look at the trailer video created completely with it.

We implemented an API to manipulate the slideshow before and during playing, we generate GUI using scripts and implemented an IDE with debugger and code completion. We implemented a powerful publishing system to export Delphi interfaces to scripting space via Attributes and RTTI.

The scripting system is intended to be used mainly by the developers and preset designers but we provide all the tools users too.

Our Interface based API can be found here (German only): http://wiki.aquasoft.de/wiki/index.php/Dokumentation_der_Scripting-API

How scripting is used:

• build intelligent presets which generate parts of the slideshow out of more complex templates.
• generate basic GUI for presets (e.g. Animated Path)
• allow execution of HLSL shaders, build their GUI and calculate their parameters
• generate context menu entries for SlideShowObjects
• manipulate bitmaps

Another product is AquaSoft WebShow 4 which is a tool to create web galleries out of photos and videos. DWScript is used here in its original manner using the HTML Filter to mix pascal code with html. Later we used more and more an HTMLBuilder class with fluent interface written completely in DWScript to avoid the html fragments mixed in the templates.

The templates are created object orientated and use inheritance. DWScript was a replacement for an own scripting language which was used in earlier versions. WebShow 4 is unfortunatelly not completely ready yet as it only has three templates included but is already public as the web wizard shipped together with AquaSoft SlideShow 8.

Eric News AquaSoft, DWS, Showcase

DWScript COM Connector has just received a spring cleanup, as it had grown a bit mouldy over the last years.  The COM Connector optionally gives scripts direct access to ActiveX and COM objects.

As part of the cleanup, it now support IEnumVARIANT and ComVariantArray can now be enumerated as well with “for in“.

In practice it means you can now use it to query WMI information f.i., as an illustration the following code will run a WQL query and print the name of the processor running the system:

const wbemFlagForwardOnly = $00000020;

var locator := CreateOleObject('WbemScripting.SWbemLocator');
var service := locator.ConnectServer('localhost', 'root\CIMV2', '', '');
var objectSet := service.ExecQuery('Select Name from Win32_Processor',
                                   'WQL', wbemFlagForwardOnly);

for var item in objectSet do
   PrintLn(item.Properties_.Item('Name').Value);

And speaking of WMI, Rodrigo RUZ maintains a lot of useful WMI queries on his The Road To Delphi blog.

Eric News, Tips COM, Connector, DWS, WMI

This is a belated news update, with only the highlights:

Language News

• combined property/fields declaration is now supported (same syntax as Oxygene)
• dynamic arrays now support a Remove method, which has the same syntax as IndexOf, and can be used to remove elements by value
• for var xxx in array syntax is now supported, which combines a local, type-inferenced variable declaration and a for … in loop.
• unit test coverage is now at 97% for the compiler, 91% for the whole of DWScript
• various obscure bugs found and fixed

Script engine News

• script engine transition from stack-based to closure-based has begun, besides internal changes, the visible impact should be improved performance for objects, records, static arrays, var and const params has been improved
• full transition to closure-based engine (and support for anonymous methods and lambdas in the script engine) is pushed back to 2.4

Teaser News

Also as way of a teaser, here is a screenshot related to something brewing in the lab… it’s from a Delphi XE app running the Mandelbrot benchmark (timings are for several runs), and the “DWScript” in that screenshot is the script engine.

Note that since this screenshot was taken, performance has improved, and the pony has learned new tricks

 

Eric News Delphi, DWS, Fractals, JIT

I’m considering setting up a DWScript showcase webpage. If you’re using DWScript and want to be on that page, please mail the following to “eric at delphitools.info”:

• short description of how DWScript is used
• screenshot of the application where it’s used
• website or product link

Usage cases don’t have to be fancy, or pretty, industrial and command-line are acceptable too! The screenshot doesn’t have to show the script, just be representative of the application.

Besides the showcase, this is also to get a feel of the variety of uses of DWScript out there, and to help figure out which way to go with future developments, so even if you don’t have anything I could post, don’t hesistate to drop a mail to let me know what you use it for.

Alternatively you can post a comment there, but to minimize spam prevention, comments don’t stay open very long on posts here.

Eric News DWS

Object Pascal does allow binding a property to a field for direct read/writer, but we all have seen properties that required a slightly more complex getter or setter, and that usually meant a method for both. DWScript (svn trunk) & Smart Pascal (1.1) now support property expressions and statements, so the syntax is extended to allow:

property Name : Type read (expression) write (expression|statement)

Under the hood, the compiler will generate an unnamed getter or setter method when appropriate.

Let’s take f.i. a class that exposes an internal list, you can now use just

TMyClass = class
   private
      FList : array of TElement;
   public
      property Items[i: Integer] : TElement read (FList[i]) 
                                            write (FList[i]); default;
      property Count : Integer read (FList.Length);
end;

Look Ma! No implementation and no constructor needed for trivial properties!
The astute reader will quickly notice how much shorter this is compared to doing it with a TList<T> and classic properties instead.

And for an hypothetical angle value types that accepts both radians and degrees you could have

TMyAngle = record
   private
      FAngle : Float;
   public
      property Radians : Float read FAngle write FAngle;
      property Degrees : Float read (RadToDeg(FAngle)) 
                               write (FAngle:=DegToRad(Value));
end;

In the setter statement, there is a special value creatively named… “Value” which holds the value assigned to the property.

In addition to expressions and statements, you can also if specify a property name as getter or setter, this will create an alias (no under-the-hood getter or setter method will be created):

TAncestor = class
   private
      FHidden : Integer;
      procedure SetSecret(v : Integer);
   protected
      property Hidden : Integer read FHidden write FHidden;
      property Secret : Integer write SetSecret;
end;

TSubClass = class (TAncestor)
   public
      property Revealed : Integer read Hidden write Secret;
      property OldReveal : Integer read Revealed write Revealed; deprecated;
end;

Which can be of use when you want to make public a protected property under a different name in a subclass, and the underlying field or method being private wasn’t visible. It can also be helpful if you want to rename a property in a migration-friendly way, with the old name deprecated.

So to summarize:

• read now accepts expressions and properties, in addition to field names and methods names
• write now accepts writable expressions (the left-side of an assignment), statements (an assignment or a method call) and properties in addition to field names and method names. “Value” contains the value assigned to the property.
• property expressions are supported for classes, records, helpers and interfaces.

As we weren’t sure of the potential ambiguity, the compiler currently requires the presence of brackets ( ) before the new expressions and statements are allowed, this requirement might be lifted in the future, or maybe not, as it makes such expressions stand out more.
This is an open question, Oxygen accepts read expressions without brackets (but doesn’t have write expressions or statements), so feedback on this aspect is welcome.

Thanks go to Primoz Gabrielcic for providing the unit tests for the above feature!

Eric News DWS, SmartMS