Wednesday, August 26, 2026

integration Lua to psql II

Ten years ago, I attempted to enhance the \dt+ command to sort results by size. There were perhaps a hundred discussions, yet no consensus was reached on a new syntax. Eventually, I created the pspg tool, which allows results to be sorted by any column based on the vertical cursor position. Now, I have prepared a set of patches that integrates Lua into psql. Thanks to these modifications, anyone can write their own \dt command with the desired behavior:
\if :{?LUA_RELEASE}
\echo :LUA_RELEASE
\luacode
psql.registerCommand ( {
  name = "my.dt",
  help_syntax = "\\my.dt[+] [PATTERN] [-OPTION]",
  help_desc = "list tables possibly sorted by size",
  handler = function(ss, ab, cmd, verbose)
    local filter = "  AND n.nspname <> 'pg_catalog'\n" ..
        "  AND n.nspname !~ '^pg_toast'\n" ..
        "  AND n.nspname <> 'information_schema'\n" ..
        "  AND pg_catalog.pg_table_is_visible(c.oid)\n"

    local sort = "ORDER BY 1, 2";

    local opt = psql.scanSlashOption(ss, psql.OT_NORMAL, false)

    if opt == "-help" then
      print "my.dt[+] [PATTERN] [-OPTION]     list tables, possibly sorted"
      print ""
      print "Options:"
      print "  -asc-size         sorted by size in ascending order"
      print "  -desc-size        sorted by size in descending order"
      return psql.PSQL_CMD_SKIP_LINE;
    end

    if opt and string.sub(opt,1,1)  ~= "-" then
      local schema, tablename, dot
      if opt == "*" then
        filter = "  AND pg_catalog.pg_table_is_visible(c.oid)\n";
      else
        dot = string.find(opt, "%.")
        if dot then
          schema = string.sub(opt, 1, dot - 1)
          tablename = string.sub(opt, dot + 1)
        else
          tablename = opt;
        end
        if schema then
          if schema ~= "*" then
            filter = "  AND n.nspname = '" .. psql.connect():escape(schema) .. "'\n"
          else
            filter = ""
          end
        else
          filter = "  AND pg_catalog.pg_table_is_visible(c.oid)\n"
        end

        if tablename then
          if tablename ~= "*" then
            filter = filter .. "  AND c.relname = '" .. psql.connect():escape(tablename) .. "'\n"
          end
        end
      end
      opt = psql.scanSlashOption(ss, psql.OT_NORMAL, false);
    end

    if opt == "-asc-size" then
      sort = "ORDER BY pg_catalog.pg_table_size(c.oid) ASC"
    elseif opt == "-desc-size" then
      sort = "ORDER BY pg_catalog.pg_table_size(c.oid) DESC"
    end

    local query = [[
SELECT n.nspname AS "Schema",
       c.relname AS "Name",
       CASE c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'm' THEN 'materialized view' WHEN 't' THEN 'TOAST' END AS "Type",
       pg_catalog.pg_get_userbyid(c.relowner) AS "Owner" ]]

    if verbose then
      query = query .. ",\n" ..
        [[
       pg_size_pretty(pg_catalog.pg_table_size(c.oid)) AS "Size",
       pg_catalog.obj_description(c.oid, 'pg_class') AS "Description" ]]
    end

    query = query .. "\n" .. [[
FROM pg_catalog.pg_class c
     LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
WHERE c.relkind IN ('r','v','m')]] .. "\n"

    query = query .. filter
    query = query .. sort

    psql.printQuery(psql.exec(query))

    return psql.PSQL_CMD_SKIP_LINE;
  end } )
\.
\endif
Usage:
(2026-08-26 22:24:29) postgres=# \my.dt+ pg_catalog.* -desc-size
┌────────────┬─────────────────────────────────┬───────┬──────────┬────────────┬─────────────┐
│   Schema   │              Name               │ Type  │  Owner   │    Size    │ Description │
╞════════════╪═════════════════════════════════╪═══════╪══════════╪════════════╪═════════════╡
│ pg_catalog │ pg_proc                         │ table │ postgres │ 912 kB     │ ∅           │
│ pg_catalog │ pg_rewrite                      │ table │ postgres │ 688 kB     │ ∅           │
│ pg_catalog │ pg_attribute                    │ table │ postgres │ 520 kB     │ ∅           │
│ pg_catalog │ pg_description                  │ table │ postgres │ 408 kB     │ ∅           │
│ pg_catalog │ pg_collation                    │ table │ postgres │ 320 kB     │ ∅           │
│ pg_catalog │ pg_statistic                    │ table │ postgres │ 272 kB     │ ∅           │
│ pg_catalog │ pg_type                         │ table │ postgres │ 168 kB     │ ∅           │
│ pg_catalog │ pg_class                        │ table │ postgres │ 160 kB     │ ∅           │
│ pg_catalog │ pg_depend                       │ table │ postgres │ 152 kB     │ ∅           │
│ pg_catalog │ pg_operator                     │ table │ postgres │ 144 kB     │ ∅           │
│ pg_catalog │ pg_amop                         │ table │ postgres │ 88 kB      │ ∅           │
│ pg_catalog │ pg_constraint                   │ table │ postgres │ 80 kB      │ ∅           │
│ pg_catalog │ pg_amproc                       │ table │ postgres │ 72 kB      │ ∅           │
│ pg_catalog │ pg_index                        │ table │ postgres │ 72 kB      │ ∅           │
│ pg_catalog │ pg_aggregate                    │ table │ postgres │ 64 kB      │ ∅           │
│ pg_catalog │ pg_ts_config_map                │ table │ postgres │ 64 kB      │ ∅           │
│ pg_catalog │ pg_init_privs                   │ table │ postgres │ 64 kB      │ ∅           │
│ pg_catalog │ pg_opclass                      │ table │ postgres │ 56 kB      │ ∅           │

Thursday, August 6, 2026

initial integration lua language to psql

It can looks like
(2026-08-06 17:44:32) postgres=# \luacode 
Enter code to be copied followed by a newline.
End with a backslash and a period on a line by itself, or an EOF signal.
>> function x(n)
>>   return n + 10
>> end
>> \.
(2026-08-06 17:45:32) postgres=# \luacode
Enter code to be copied followed by a newline.
End with a backslash and a period on a line by itself, or an EOF signal.
>> print (x(10))
>> \.
20

https://github.com/okbob/lua-psql

Thursday, April 2, 2026

Using non ACID storage as workaround instead missing autonomous transactions

When I was younger, the culture war (in my bubble) was about transactional versus non-transactional engines, Postgres versus MySQL (MyISAM). Surely, I preferred the transactional concept. Data integrity and crash safety is super important.  But it is not without costs. It was visible 30 years ago, when MySQL was a super fast and PostgreSQL super slow database. Today on more powerful computers it is visible too, not too strong, but still it is visible. And we still use non-transactional storages a lot of - applications logs. 

 There are some cases when performance wins over consistency, and it can be acceptable. When I thought about non-transactional storages, I got one idea. It can be great replacement for missing autonomous transactions. But how to test it. Fortunately I found a csv_tam storage implemented by Alexey Gordeev.  This storage is mostly a concept with a lot of limits. But the idea is great - csv is a strong protocol - it is not block based, it has no row headers - so it can be very hard to support transactions. On second hand, it is primitive, and without any buffering and with forcing syncing after any row, it is mostly crash safe (against Postgres crash). Sure - it is not as safe as block storage ensured by WAL, but can be safe enough - billions applications use this safety for logging today. 

I did fork and fixed build on pg 17+. Now all types are supported and writing from parallel writes should be safe. It doesn't write to WAL, so these tables cannot be backuped and cannot be replicated - what can be a nice game to support it. It is not easy to do that in a non-block format. But for testing it is enough, and I believe so this extension is very simple, so it is enough for non critical environments. It is really very very simple. 

Postgres has not autonomous transactions. There are some workarounds like using dblink or pg_background.  As usual any workaround has some disadvantages and limits. pg_background looks good, but at the end, it doesn't ensure 100% success in write (under high load) - although there will be space on IO. So I wrote another workaround - using a non transactional engine. Not all transactional engines are not same. If I remember well, MyISAM is non-transactional and non crash safe. Aria engine is non-transactional, but crash safe. csv_tam storage is non transactional and mostly crash safe. For fully crash safety it needs fault tolerant reading (which is now possible, and should not be too hard to implement). csv_tam supports only inserts, and truncating. Nothing more. Thanks to this it is mostly crash safe.

(2026-04-03 07:44:37) postgres=# create extension csv_tam ;
CREATE EXTENSION

(2026-04-03 07:47:10) postgres=# create table log(ts timestamp with time zone, message varchar);
CREATE TABLE

(2026-04-03 07:52:22) postgres=# \sf foo
CREATE OR REPLACE FUNCTION public.foo(integer)
 RETURNS integer
 LANGUAGE plpgsql
AS $function$
begin
  return 0/$1;
exception when others then
  insert into log values(current_timestamp, sqlerrm);
  raise; -- reraise error
end;
$function$

(2026-04-03 07:49:15) postgres=# select foo(0);
ERROR:  division by zero
CONTEXT:  PL/pgSQL function foo(integer) line 3 at RETURN
(2026-04-03 07:49:20) postgres=# select * from log;
┌────┬─────────┐
│ ts │ message │
╞════╪═════════╡
└────┴─────────┘
(0 rows)

-- it doesn't work because we used classic heap (transactional) storage

(2026-04-03 07:49:51) postgres=# create table log(ts timestamp with time zone, message varchar) using csv_tam;
CREATE TABLE

(2026-04-03 07:49:58) postgres=# select foo(0);
ERROR:  division by zero
CONTEXT:  PL/pgSQL function foo(integer) line 3 at RETURN
(2026-04-03 07:50:01) postgres=# select * from log;
┌───────────────────────────────┬──────────────────┐
│              ts               │     message      │
╞═══════════════════════════════╪══════════════════╡
│ 2026-04-03 07:50:01.437296+02 │ division by zero │
└───────────────────────────────┴──────────────────┘
(1 row)

Friday, December 19, 2025

fresh dll of orafce and plpgsql_check for PostgreSQL 17 and PostgreSQL 18

I compiled and uploaded zip files with latest orafce and plpgsql_check for PostgreSQL 17 and PostgreSQL 18 - I used Microsoft Visual C 2022.

Setup:

  1. download orafce-4.16.3-x86_64-windows.zip or plpgsql_check-2.8.5-x86_64-windows.zip and extract files
  2. copy related dll file to PostgreSQL lib directory (NN is number of pg release)
    orafce-NN.dll -> "c:\Program Files\PostgreSQL\NN\lib"
  3. remove suffix "x64-NN" from dll file
    orafce-NN.dll -> orafce.dll
  4. copy *.sql and *.control files to extension directory
    *.sql, *.control -> "c:\Program Files\PostgreSQL\NN\share\extension"
  5. execute with super user rights SQL command CREATE EXTENSION
    CREATE EXTENSION orafce;
     

Note: plpgsql_check and Orafce are noncommercial extensions for PostgreSQL. These extensions are available on AVS, Azure, Google Cloud. It can be easy installed on Linux from community repositories, but these repositories has not build for Windows.



Saturday, April 19, 2025

Article about PostgreSQL 18

I wrote an article about PostgreSQL 18.  It is in Czech language, but translators from Czech to English, German, ... works relatively well today.

 https://www.root.cz/clanky/postgresql-18-tricet-let-otevreneho-vyvoje-databaze/

Saturday, March 29, 2025

How to fix Hibernate bug by conditional index

Yesterday I found significant grow of seq read tuples. After some investigation I found query with strange predicate:

WHERE 1 = case when pb1_0.parent_id is not null then 0 end

It is really strange, and I had to ask, who wrote it.

The reply is - Hibernate. It is a transformation of predicate parent_id = (?) when the list of id is empty.

Unfortunately, PostgreSQL is not able to detect so this predicate is always false, and then the repeated execution ended in repeated full scans.

Fortunately, Postgres has simple workaround - conditional index

CREATE INDEX ON TABLE pb(id)
  WHERE 1 = case when pb1_0.parent_id is not null then 0 end

This index is always empty, and then index scan is fast.

This issue should be fixed in more recent versions of Hibernate where predicate 1=0 is generated instead.

Thursday, February 27, 2025

fresh plpgsql_check 2.7.15 for PostgreSQL 16, 17 for MS Windows

 

I compiled and uploaded zip files plpgsql_check for PostgreSQL 16 and PostgreSQL 17 - I used Microsoft Visual C 2022.

Setup:

  1. download plpgsql_check-2.7.15-x86_64-windows.zip and extract files 
  2. copy related dll file to PostgreSQL lib directory (NN is number of pg release)
    plpgsql_check_NN.dll
    -> "c:\Program Files\PostgreSQL\NN\lib"
  3. copy *.sql and *.control files to extension directory (the version number) should be removed.
    *.sql, *.control -> "c:\Program Files\PostgreSQL\NN\share\extension"
  4. execute with super user rights SQL command CREATE EXTENSION
    CREATE EXTENSION plpgsql_check;