Caching Options

Optional caching with LMDB or Redis.

Warp uses LevelDB on Node and IndexedDB on browser for caching by default. It allows overriding the cache to be used via useStateCache and useContractCache; as well as overriding the underlying key-value storage via useKVStorageFactory. These overrides accept any cache interface that supports the SortKeyCache interface of Warp Contracts, see the relevant documentation here.

Since HollowDB takes input a warp instance, by applying cache overrides to the warp instance outside and passing that instance to HollowDB, you will be able to use the caching of your choice for HollowDB!

LMDB Cache

LMDB is an ultra-fast NodeJS, Bun, and Deno interface to LMDB; probably the fastest and most efficient key-value/database interface that exists for storage and retrieval of structured JS data (objects, arrays, etc.) in a true persisted, scalable, ACID compliant database

Warp Contracts provide an LMDB cache interface under warp-contracts-lmdb package. It can used as follows:

import {defaultCacheOptions, WarpFactory} from 'warp-contracts';
import {LmdbCache} from 'warp-contracts-lmdb';

warp = WarpFactory
  .forMainnet()
  .useStateCache(
    new LmdbCache(
      {
        ...defaultCacheOptions,
        dbLocation: './cache/warp/state',
      }
    )
  )
  .useContractCache(
    new LmdbCache({
      ...defaultCacheOptions,
      dbLocation: './cache/warp/contract',
    }),
    new LmdbCache({
      ...defaultCacheOptions,
      dbLocation: './cache/warp/src',
    })
  )
  .useKVStorageFactory(
    (contractTxId: string) =>
      new LmdbCache({
        ...defaultCacheOptions,
        dbLocation: `./cache/warp/kv/lmdb_2/${contractTxId}`,
      })
  );

Redis Cache

Redis is an open source (BSD licensed), in-memory data structure store used as a database, cache, message broker, and streaming engine. Redis provides data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes, and streams.

We have prepared a Redis SortKeyCache implementation under warp-contracts-redis. It can be used as follows:

import {WarpFactory, CacheOptions} from 'warp-contracts';
import {RedisCache, RedisOptions} from 'warp-contracts-redis';

// in case you might use this Redis for multiple contracts,
// it might be best to have a different key for each contract!
const contractTxId = "your-contract-tx-id";

const cacheOptions: CacheOptions = {
  inMemory: true,
  subLevelSeparator: "|",
  dbLocation: "", // we will override this
};
const redisOptions: RedisOptions = {
  url: constants.REDIS_URL,
};
warp = warp
  .forMainnet()
  .useStateCache(
    new RedisCache(
      {
        ...cacheOptions,
        dbLocation: `${contractTxId}.state`,
      },
      redisOptions
    )
  )
  .useContractCache(
    new RedisCache(
      {
        ...cacheOptions,
        dbLocation: `${contractTxId}.contract`,
      },
      redisOptions
    ),
    new RedisCache(
      {
        ...cacheOptions,
        dbLocation: `${contractTxId}.src`,
      },
      redisOptions
    )
  )
  .useKVStorageFactory(
    (contractTxId: string) =>
      new RedisCache(
        {
          ...cacheOptions,
          dbLocation: `${contractTxId}.kv`,
        },
        redisOptions
      )
  );

Redis is not exactly like the other cache options (LMDB or the default LevelDB) which are "local". In the case of Redis, we can host a Redis server and connect to it, thereby solving the problem of downloading the entire state on different machines each time. We can also use the same client on multiple cache types, for example a single Redis client in our application can be used for state cache, contract cache and kv-cache at once.

For this purpose, we allow the user to create the client outside, and pass it to the RedisCache in the constructor. Here is an example using ioredis package for the client.

// create client
const redisClient = new Redis("connection url", {lazyConnect: true});

// pass the client in constructor
warp = warp
  .useKVStorageFactory(
    (contractTxId: string) =>
      new RedisCache({
          ...cacheOptions,
          dbLocation: `${contractTxId}.kv`,
        },
        { client: redisClient }
      )
  );

// define custom scripts used by RedisCache
RedisCache.defineLuaScripts(client);

// connect to client manually
await redisClient.connect();

// optional: disable persistent memory
// this is what `inMemory: true` normally does
await Redis.setConfigForInMemory(client);

The extra steps in the end are normally done internally, but to avoid repeating them in consequent usages of the same client, we expect the user to take the responsibility of doing them. A warning log also notifies the user of this at runtime.