Sneak Preview

By Randy Merilatt, Raima CTO

It is not really any secret that we currently have under development a brand new SQL for our RDM Embedded (RDMe) product that is due out around the end of the first quarter this year. I have been the principal architect for the new SQL and that should be my final project as a software developer here at Raima. I am of course excited about taking on the role of CTO but at the same time, I am enjoying the building a new SQL system that is designed for the needs of the embedded systems software developer. I will be presenting the “Elephant” paper on the new RDMe SQL (“Is Using SQL in an Embedded Computer Application Like Trying to Squeeze an Elephant into a Mini?”) at the Embedded World Conference in Nürnberg, Germany on March 3rd. If you’re planning to attend, please let me know so we can set up a time to meet.

Raima’s first DBMS product, released over 25 years ago, was called db_VISTA. Our first venture into the world of SQL came in the form of db_QUERY which was an SQL-like query and report writing tool for db_VISTA databases. In 1992, we introduced our client/server DBMS originally called Raima Database Server (RDMs), shortly thereafter renamed Velocis and now called RDM Server. The client/server architecture was designed for resource-rich computers allowing us to provide a full-featured SQL implementation. Today, except perhaps for the smallest of embedded computing devices, most computers have sufficient computing resources to support a healthy subset of SQL and so we will soon be providing an SQL constructed on top of the RDMe database engine.

The native API for RDMs SQL is based on ODBC. However, ODBC is designed for a wide variety of SQL systems. Besides that, it is inconsistent and its specification is ambiguous in places. It also tends to be a moving target as changes continue to be made to it. Because of this, we decided that we would design our own native API for RDMe SQL. It is not ODBC but it is similar enough so that an experienced ODBC programmer will have no difficulty in learning it. For example, function rsqlPrepare compiles as statement specified in a null terminated string as does SQLPrepare but without having to pass SQL_NTS. Data type codes are defined using a enum typedef instead of ODBC’s #define constants which are subject to change over time (e.g., SQL_DATE, etc.). This native API is more efficient in that it only needs to deal with the specific features provided in RDMe SQL. And, because this API is not ODBC, the SQL runtime engine implementation is simpler.

An ODBC API is also provided as a layer over the native API. So, if use of ODBC is required it can be used either through direct link with the RDMe libraries and/or through client/server access to the RDMe SQL database from a remote computer.

Besides providing SQL, this upcoming release of RDM Embedded will also sport a new C++ API which will be discussed in an upcoming blog..

I am excited about the new Raima. While we are grateful for the support received from Birdstep over the last decade but because we are now independent we have a steadfast commitment only to the database products and our customers. We have the freedom to dream and to execute on those dreams. We have a wonderful spirit of unity and commitment among all of the management and employees here at Raima. Our goal is to have highly satisfied customers and to enjoy the experience of making that happen together.

RDM Embedded's Transactional File System

There are many programmers who, over the past 20 years, have implemented a multi-user database system using db_VISTA, Raima Database Manager or RDM Embedded (depending on when you got on board). Congratulations! That was not easy.

The next release of RDM Embedded has been re-architected for efficient, scalable multi-core operation, but it has also become much simpler to use. It turns out that application developers don't want or need to configure every little option in order to use a database. Rather, they would just like to use it. That may seem like a subtle point, but it has had a dramatic effect on usability of RDM Embedded by the programmer.

Both usability and multi-core awareness are facilitated through the new component of RDM Embedded, called the Transactional File Server (or TFS, as I will refer to it). The TFS owns databases, controlling how they are read and updated. In the old architecture, the runtime library did all this from within each application that was using an Embedded database. The TFS centralizes the critical functionality of database reading and updating. It is the central piece in satisfying the major multi-core design principle I discussed in the last blog entry:

• Obtain all resources needed for a task (the TFS requires database pages to be locked prior to reading, then it provides the pages to the requesting runtime libraray).
• Perform the task. This is the job of the runtime library. After obtaining existing database pages, or creating new ones, the runtime library will submit a completed transaction to the TFS to commit the transaction.
• Quickly merge, or re-integrate the results of the task. Since the runtime library has encapsulated the changes of a transaction into a single unit (actually, a log file), the TFS has a minimal amount of work to do to commit the changes. The "heavy lifting" was done by the process linked to the runtime library, and the TFS has little to do but flip the "commit" switch.

So the way it all fits together is like this. The runtime library is linked into application programs. As always, the runtime library reads database pages, makes changes to them, creates new pages, then writes out all pages, modified or new, to the database. Each application in the system has its own copy of the runtime library maintaining its own local cache and submitting the new or changed pages to the database. Only now, the reading is through the TFS and the writing is through the TFS. In all of this, the real work is done by the runtime library in its local cache. The TFS, serving pages and committing transactions for multiple runtimes, is left with relatively little to do.

This allows multiple cores to work simultaneously without blocking each other. Even within the TFS, there are threads that service the runtimes which are mostly independent.

Now for the usability claim. The TFS is the runtime's only "connection" with the outside world - if it connects with the TFS, it can do its job. Connection is established through the most pervasive mechanism ever known - TCP/IP. If the TFS is visible, the runtime can connect to it. There is no more shared disk drives or TRUENAME for identifying files, no lock manager, DBDPATH, DBFPATH, DBTAF, ... (ugh, what were we thinking at the time?).

So, a TFS running on a computer does not need to share any files or drives in order for other runtimes to use the databases within the TFS domain. Runtimes may be on the same computer as the TFS, another computer on its LAN, or somewhere else on Internet. The only difference will be performance, not whether it works or not.

This brings up an unintended benefit of this architecture - not only can multiple cores on one computer allow the TFS and runtimes to scale up together, but other computers can be added to the mix. Multiple cores plus multiple computers all working together in parallel. Not only that, but one runtime can access databases from multiple TFS's. Not only that, but one runtime can view databases scattered throughout the world in different TFS's as though they are one unified database.

The separation of database semantics and operations (the runtime) from the TFS (safe, transactional updating of files) lets the pieces fit where they need to fit, and to be combined in an indefinite number of ways, depending on the need.

Usability now means more than "easier to use," it means usable in ways never considered before.

Mid-Course Correction, Again

In 1991, Raima sold a linkable C library called db_VISTA, which worked fine for small work-groups that needed to share a database, but the database industry in general was more interested in the client/server architecture.

So, we started with db_VISTA and created a database server which was named Velocis. We had two products, aimed at two types of applications. This was the first major mid-course correction. Technically, it involved a fundamental architectural change. db_VISTA performs disk I/O directly from each application program, keeping a local cache of database pages. Velocis performs disk I/O only from the server, and keeps only one centralized cache.

Fast-forward to 2010. db_VISTA is now called RDM Embedded, Velocis is now called RDM Server, and they are both actively used and supported. The company has gone through multiple ownership changes, but the products have remained viable for over 20 years. Raima is now a division of Birdstep Technology. The original developers (Randy Merilatt and me, Wayne Warren) are back after trying other things for a while, and other long-term developers have survived all the changes. The products have had several new features added, run on new platforms, and are much more reliable.

But the industry has changed, and another mid-course correction was necessary. To state some of the obvious changes:

• Memory is massive and cheap,
• Disk space is massive and cheap,
• Networking is fast and pervasive, and
• Multi-core and multiprocessor systems are common.

It turns out that a database system that was architected to optimize memory and disk and a single CPU does not scale up as one would expect it to. This is not good, but it is something that is being realized throughout the industry. Multi-core scaling requires an architecture that turns each core loose, rather than each core interfering with the others. I'll talk more about a multi-core-aware architecture in another post.

This second major mid-course correction started again from the smaller product, RDM Embedded, and will be released as the next version of RDM Embedded. It will work for the applications currently using RDM Embedded, but the underlying architecture has changed to take advantage of all of the above - use memory to obtain speed, use networking to achieve parallelism, and program the database access to allow multiple cores/processors to run in parallel.

It turns out that by doing all of the above, the product also became simpler to use and deploy, and more flexible in using multiple computers together. You just start using it, and it just starts working (any of you old-school RDM Embedded programmers will love this).

I'll get into more details as I have time. But I'm also updating the documentation, which is keeping me very busy.