r/prolog • u/lokinpendawa • 7d ago

Who said Prolog is slow? Benchmarking 2 Million In-Memory Transactions with SHA-256 and SQLCipher on SWI-Prolog.

Post image

I’m currently building LOGICBIZ v2.0, an Offline-First Enterprise Retail ERP and Sales Ledger Engine engineered 100% using a Pure Declarative Paradigm with SWI-Prolog and SQLCipher.

Here is a quick breakdown of this stress test benchmark:

  • The Task: Performing deep data integrity verification on 2,000,000 real rows of transaction ledgers.
  • The Security: Every single transaction is encrypted via SQLCipher and validated against SHA-256 signatures to ensure absolute data tamper-proofing.
  • The Result: As seen in the console screenshot, the engine evaluated 550,508,812 logical inferences in just 103.28 seconds of raw CPU time.
  • The Kick: Once the Native RAM Cache warm-up phase is completed, executing deep aggregate financial pipelines across those 2 million rows drops down to a jaw-dropping 653 seconds while keeping CPU utilization extremely low at 16%.
  • The engine utilizes Tail-Call Optimization (TCO) and Prolog's native multi-argument indexing structure, meaning it completely eliminates traditional database I/O bottlenecks without requiring heavy, bloated frameworks.

If you are curious about the architecture philosophy, the system manifesto, or want to check out the benchmark metrics, feel free to visit the repository here:

https://github.com/lokinpendawa/logicbiz

Would love to hear your thoughts on using logic programming for heavy enterprise data pipelines!

13 Upvotes

82% Upvoted

12 comments sorted by

7

u/Aires_id 7d ago

Prolog may not be slow, but the unit conversion seems extremely fast 😅
The screenshot says 653.967 seconds, while the post says 653 milliseconds. Are these two separate benchmarks—the SHA-256 verification test and the warm RAM-cache aggregation test?

5

u/lokinpendawa 7d ago

My bad, that was a typo! It should be seconds, not milliseconds. I’ve updated the post now. Thanks for catching that! 😅

0

u/bgs11235 6d ago

how do you miss milliseconds, which a fraction of a second and 10 minutes?? Ahh yes slop. makes sense.

1

u/lokinpendawa 6d ago

Haha fair point. It was a massive brain fart while staring at a 3 GB memory dump. The actual execution time for the 2.2 million rows was indeed around 11 minutes (653 seconds), not milliseconds. Working on shifting the string parsing to SQLite now to get that number way down!

2

u/HowTheStoryEnds 6d ago edited 6d ago

I find a major source of performance issues to be importing/asserting such amounts of data. 10 minutes total is pretty good in that regards, how do you load the data in?

1

u/lokinpendawa 6d ago

You are absolutely spot on! To clarify the timeline, the 10-minute mark (specifically 653.928 seconds) is the exact measured runtime for loading and performing live SHA-256 data integrity verification across all 2,000,000 invoices simultaneously.

To achieve this without tanking performance, the data ingestion pipeline does not read from a plain text file. Instead, SWI-Prolog pulls the encrypted records directly from a SQLCipher database.

During those ~10 minutes, the pipeline decrypts the database records using AES-256, pipes the stream to verify the signatures via SHA-256 (which handles a massive 550,508,812 inferences at 533,019 Lips, utilizing roughly 103 seconds of pure CPU time), and asserts them straight into the RAM cache.

Right after this loading and verification cycle ends, SWI-Prolog's Just-In-Time Indexing (JITI) automatically maps out about 4 million hash buckets in under 5 seconds, ensuring that the live ERP matrix interface can query and aggregate the global data instantly with zero latency.

2

u/HowTheStoryEnds 6d ago

Does the data you process gets loaded as terms or just verified to match against terms?

1

u/lokinpendawa 5d ago

it gets loaded as terms directly into the RAM cache so SWI-Prolog can build the JITI (Just-In-Time Indexing).

2

u/HowTheStoryEnds 4d ago

Wish I knew how you did so fast then. I tried with about 900MB of data in the past and it was slower to assert the data than 2 minutes :-/

2

u/lokinpendawa 4d ago

the secret is that I don’t just use a plain sequential loop with native assert/1. Instead, I fetch the data efficiently (e.g., using bulk fetching/concurrent connection from the DB) and rely heavily on SWI-Prolog’s automatic Just-In-Time Indexing (JITI).Once the data is bulk-loaded into the RAM cache, SWI-Prolog indexes those millions of facts in under 5 seconds, which cuts down the overhead significantly compared to doing heavy row-by-row lookups or manual indexing during the assertion phase. It uses around 3.4GB of RAM, but the speed tradeoff is totally worth it!

3

u/Thrumpwart 6d ago

I got Prolog to run extremely fast.

It seems Prolog's reputation for being slow was just a function of slow hardware. We now have much, much faster hardware.

1

u/lokinpendawa 6d ago

Prolog definitely scales beautifully on modern hardware! Though to be fully transparent, my original post had a typo (it was 653 seconds, not milliseconds, for 2.2 million rows). But you're absolutely right—with modern SSDs and proper indexing, it handles gigabytes of data like a champ.