Blackfish SQL part 2
In my second blog on Blackfish SQL I have highlighted several of its key features. Many of these features are what make this product easy to work with and easy to deploy.
Simple deployment.
There are single file deployment options for both the .NET and Java platforms. If you use the windows version, there is only one .NET assembly you need to deploy. This is particularly nice for ASP.NET applications. No longer need to rent a database from your ISP. On the Java platform there is a single jar file to deploy. The license file also needs to be deployed, but this is small and it is “not” machine locked. This makes it easy to do simple xcopy deployments. On both platforms the delployment size of the binaries is about 1.3 megabytes.
Industry compliance.
SQL-92 compliant. Local and remote ADO.NET providers. Local and remote JDBC drivers. DBX4 100% Object Pascal win32/.NET driver.
Performance.
Early on we would do benchmarks against other Java databases. However we quickly found that we were orders of magnitude faster than them. We also found that we had no problem competing with databases written in native code. Invariably, every time we found ourselves being beat it had nothing to do with managed code vs native code. We could always match or beat the competition with better algorithms. I know native code can be made faster for simple code, but with the complexity of transactional, high concurrency database kernel, better algorithms are king. We have not performed major tuning for a while, and I’m not going to claim it’s the fastest, but this database is fast.
In-process execution.
The local ADO.NET and JDBC “local” drivers execute in process for improved performance and ease of deployment.
Delphi, C#, VB.NET, and Java stored procedures and triggers.
These are safer and more powerful languages to write database stored procedures and triggers in. In addition to using mainstream languages, you can also use off the shelf IDEs, debuggers and profilers to develop your stored procedures and triggers.
Really cross platform.
Blackfish SQL pretty much runs every where .NET and Java run. We also have built a compact framework assembly for Blackfish SQL for the compact framework as well. We are not ready to release this now, but we know it can be done.
Unicode.
This is the only way Blackfish ever knew to handle character data. Dealing with character data is always simple. No multi-byte, national character, etc issues. You also don’t need to specify the length of a string for performance reasons. String storage is transparently managed. For collation we use the collation key support provided by the .NET and Java platforms. Why reinvent the wheel?
64 bit support.
We didn’t have to lift a finger for this support. If the .NET virtual machine is 64 bit, you can have a 64 bit database cache. The same for a 64 bit Java vm. Same binary for both 32 and 64 bit platforms.
Security.
Databases can be password protected and there is SQL based security including support for oles. The database file itself can be encrypted using the Two fish 128 bit Two Fish block cipher. Don’t forget your password. There is no back door. Some security conscious companies and government agencies view 100% managed code solutions as more secure.
BDE.
We have mentioned in our roadmap that Blackfish SQL as a BDE replacement. Today this is still a bit of work because it requires that you move away from the simple TTable/Query navigational data access model. Going forward we will be looking at what we can do to support a navigational model to facilitate BDE application migration to Blackfish SQL. The idea is to support native win32/64 client applications that can connect to a Blackfish SQL server using a navigational data access model (TTable).
Tooling.
Our Blackfish SQL tooling is very basic. Leonel put a lot of work into adding ADO.NET, dbExpress 4 and query builder support into the DataExplorer in a relatively short period of time. Our plan is to improve the capabilities and extensibility of DataExplorer going forward not just for Blackfish SQL, but also for Interbase and the rest of the databases we support as well. In the mean time, Blackfish SQL has built-in “DB_ADMIN” stored procedures that allow applications to perform many common administrative tasks including database backup/copy/defragment/verify/encrypt/configuration, change password, metadata retrieval, active connections, lock manager state, and server side logging configuration. The built-in DB_ADMIN stored process allows us to avoid polluting the name space of our standards based SQL dialect with Blackfish SQL specific extensions.
Share This | Email this page to a friend
Posted by Steve Shaughnessy on September 10th, 2007 under Uncategorized |
RSS Feed
September 11th, 2007 at 12:45 am
Congrats on your new DBEngine. Seems interesting, specially the built-in encryption. I don’t know if I’ll ever choose it over (DI)SQLite, but it’s always good to know we have options.
There comes some questions to mind:
1- Will Blackfish SQL be available in Native code for Delphi? If so, when?
2- Is it single file?
3- What are the limits for Table size, BLOB size and DB file(s) size?
4- Is the DB file(s) portable between Big-Endian and Little-Endian machines?
5- Still can’t figure why you are going to change BDE with BFSQL! DBE is native code, and somehow embedded (not in a good way, but hey!). Why change it with a managed code solution? Also, I hope you use some kind of dbExpress wrapped components as replacement! The only guess I have is because you’re going to heavily invest in VCL.NET, which is a good idea IMHO, but doesn’t upgrade the tools for win32, where most people use Delphi for.
Also it won’t be a bad idea to have a look at DBWorkbench. It is very nice, supports many DB engines, and could easily support BFSQL.
It’ll be very nice to license a very limited version for RAD Studio, so we can finally get a good tool to manage our databases in a decent way!
And an idea:
While you’re at it, please add an effective compression option to DB file(s) or at least BLOB fields.
September 11th, 2007 at 12:48 am
Very impressive. Bravo!
September 11th, 2007 at 9:57 am
If this thing is all you say it is, what does it mean to the future of InterBase?
Why would CodeGear want to continue to develop a 20 year old product they didn’t originally write, whose source code is still in C (assuming you didn’t port it to C++?), that has an open-source, "free as in beer" version available, when you now have a modern, managed-code database engine that you will be able to evolve much more quickly?
Wouldn’t it make sense to re-open-source the latest version of IB, let the Firebird project continue its development to keep your existing customers happy, and continue to sell support for it but not licenses?
September 11th, 2007 at 10:13 am
KA:
1) We don’t currently plan to make the server available in native Delphi. It’s a possibility, but there are higher priority feature requests.
2) The database it self is single file. However the transaction log files are separate for performance and fault tolerance. Log files are automatically dropped when no longer needed and can be truncated to reduce deployment size.
3) Max database size is over 8 terabytes for a 4k block size. Max rows is 281,474,976,710,656, Max blob size is 2GB.
4) File format is portable. There are no big/little endian issues. However there are some minor differences between the .net and java servers. If you have an object column, the serialization for .net and java are incompatible. The Java timestamp and decimal are a little higher resolution than .net.
5) Using Blackfish SQL as a BDE replacement still allows the client to be 100% native. Note that the new DbxClient that can connection a Blackfish SQL server is already 100% Object Pascal and can be compiled for both win32 and .net. We don’t currently have support for TTable/Query semantics used by BDE applications.
As for DBWorkBench, we don’t expect or want to replace all database tools. However we will continue to expand our tooling infrastructure in terms of capability and extensibility going forward. The new dbExpress 4 metadata we just rolled out is an important step for us, but I still view this as a primitive layer for our tooling infrastructure.
Support for compressing blobs could be a good feature depending on the content.
September 11th, 2007 at 10:13 am
K A –
I’ll try to answer your questions:
There comes some questions to mind:
>>1- Will Blackfish SQL be available in >>Native code for Delphi? If so, when?
No — this is a managed solution. We have no plans to offer a native version. If you need a native database, I’d suggest looking at Interbase.
> 2- Is it single file?
Yes. One file, about 1.3MB in size.
>>3- What are the limits for Table size, BLOB >>size and DB file(s) size?
Steve can answer that better than I, but I believe that all of them are ridiculously large.
>>4- Is the DB file(s) portable between >>Big-Endian and Little-Endian machines?
I’ll leave that one to Steve as well.
Nick Hodges
Delphi Product Manager
September 11th, 2007 at 10:50 am
Nick
The offering for Architect users with limiting size of 2GB is not very attractive.
Vistadb available is $279(with SA) royalty free with support for # In-memory databases and tables # XML import and export # Secure Blowfish encryption (32-bit to 448-bit) # NET Framework 1.1,2.0,3.0 -. Oracle Express limit is 4Gb. I am sure there are more.
I do not find Blackfish attractive for the above reasons.
September 11th, 2007 at 10:58 am
Th:
1) Delployment. File format is portable. There are no big/little endian issues. However there are some minor differences between the .net and java servers. If you have an object column, the serialization for .net and java are incompatible. The Java timestamp and decimal are a little higher resolution than .net.
2) Performance. I’m not going to mention specific databases that we tested against because 1) Our tests are invariably biased. 2) I don’t want to have a lawyer tell me that I violated some one’s license agreement. My main point is that a managed code database can have performance that is competitive with the performance of a database written in native code. On 2.5 I am typing too fast.
3) A classical log based transaction management model is employed. The design is heavily influenced by Jim Gray and Andrea’s Reuters text on transaction management. Even when using the simplest designs, transaction management and concurrency control for databases is one of the most complex software problems. The biggest value I see in a multi-generational architecture is support for long duration read only transactions. Blackfish SQL has support for long duration read only transactions that do not acquire locks from the lock manager. A transactionally consistent view of the database is provided when a read only transaction is started. The Blackfish SQL read only transaction support is non-trivial, but it is still less complex than a MVCC approach and does not have side affects such as version management inside the database itself and garbage collection. As for BDE, I worked on Paradox DOS database internals years ago and I confess that I am responsible for the lock file implementation. This is a terrible way to implement lock management, but we could only rely on file servers back then. I have experience with the navigational model and the weak concurrency model often employed. Also note that the internals of Blackfish SQL have been influenced by my work on DOS Paradox. In particular, support for a navigational access model.
4) Tooling. The new dbExpress 4 metadata is a big step for us. There is one more layer I want to put on top of this that will be very powerful for a variety of tooling needs including database consoles, schema designers, database factoring, isql, etc.
September 11th, 2007 at 11:08 am
Steven Summers:
Interbase is a healthy business with a lot of customers that love it. The current devleopment team has been in tact for many years. I know at least one of the key developers has been working on it since the 80s. There are lots of databases in the world. There does not have to be just one at Codegear. I think Sybase has at least 3. While there is overlap in the applications that can use either Interbase or Blackfish SQL, they do have differing strengths. There are roadmap updates and FAQs in progress that I think will make this more clear.
September 11th, 2007 at 11:37 pm
Steve,
Unfortunately, changing blog engine has resulted in comment loss (Or so it seems)
Not only I can not see my own previous comment, but your comment has been truncated either.
Is there a way to fix this? I have lost an important reply of yours and I think I may need that info.
Thank you.
September 12th, 2007 at 10:17 pm
Steve,
Thanks for detailed info, and quick response.
Just one other __VERY IMPORTANT__ question: How exactly a managed code database can be embedded into a win32 application?!
Is CodeGear going to develop hybrid solutions for win32 and .NET?
The only reason we love Delphi is because it is self-contained. If you are planning on hybrid solutions, please let us know before hand, so that we can make strategic decisions.
This will have a HUGE impact on the community.
September 14th, 2007 at 12:12 am
Steve, istm that you have a problem with the blog engine. The blog answers are truncated for your last (three - AFAIS) blog posts. It’s really too bad. Now when finally you found the time to write something…
Can you pester John, please? Don’t worry, I’ll do also my part…
September 17th, 2007 at 10:37 am
Nick:
Can Interbase run embedded or is it only a local client server model.
I understand Firebird can run embedded but apparently there is no DBX4 driver for that (as far as I know)
So if I want to run a native embedded DB application thats DBX4 compliant what would you suggest?
October 1st, 2007 at 2:41 pm
So why do you guys do another Database? Why not stick with Interbase, FireBird?
November 5th, 2007 at 12:15 pm
Steve,
Sorry by off-topic.
I have interest in make a DBX4 driver for Firebird/pgSQL.
Can I contact you by e-mail?
March 9th, 2008 at 6:14 pm
two quick questions:
1. how could a managed sever be fast?
well, i know, you will say "binary search is not fast because of the compiler"
2. in your user guide (pdf) somehow intrusively place on the root of delphi 2007 dvd you mentioned about row level locking on inserts… that’s a very good one! (but i guess you were talking about triggers…)
anyway, good luck!
ps
did you hear about sqlite? seams that everybody’s using it! for similar purposes (desktop use). when it comes to servers, my recommendation to you would be to:
a. take a look at postgresql (a little bit of oracle would not hurt either)
b. there is a nice book about database theory it would be nice if you opened. it’s call transaction processing concepts and techniques by jim gray and anders reuter
best regards,
anonymous
May 8th, 2008 at 7:28 pm
>However we quickly found that we were orders of magnitude faster than them.
architects design algorithms and apply design patterns KNOWING in advance their BIG O notation. it’s funny that you "quickly found out" how fast you are by benchmarking an already built product
lol! is this more like "qa" kind of engineering you practice there (psst, don’t tell embarcadero folks…)
>Many of these features are what make this product easy to >work with and easy to deploy.
you forgot to mention how "affordable" the product is (to deploy) - thinking at comparing it with sqlite, mssql express editions, etc
>The idea is to support native win32/64 client applications >that can connect to a Blackfish SQL server using a >navigational data access model (TTable).
right, ttable was the best of bde
lol! actually, if you could implement the paradox sliding buffer and smart indexes, this would be cool, but guess what, the sql nature of your back end might turn this into a decades long marathon, no worries here eighter (all customers are using 3rd party vcl components for years by now) - i wonder what embarcadero’s folks (quite knoledgeable in database area!) have to say about the subtle innovation in delphi databases vcl… 
>Unicode. This is the only way Blackfish ever knew to >handle character data
great again! the worst choice in town, any chance for utf-8? nope, no infrastructure, ops, sorry
>We didn’t have to lift a finger for this support.
that’s one of the funniest one! trully! that’s why we provided x86 and x64 targets in visual studio, but sure, don’t bother lifting a finger
lol, cannot wait to see the embarcadero folks when you’re gonna present them this vision!
>Really cross platform
we are so happy to learn that you support all windows platforms!
lol again
have fun, there is a lot of fun, just around the corner!
September 10th, 2008 at 6:08 am
Hi Steve, i can`t find any examples, how to use BSQL "In-Process". Where can i get sample projects and more information (except Blackfish™ SQL Developer’s Guide) about complex using BSQL+Delphi.Net+VCL.NET+Reports?
Thank