Tuesday, March 10, 2009

Experimental psql

Hello

I spent some on psql's face lifting. Some new features are really experimental. I don't expect early adopting it for core. It's more game for now. epsql will be available with release 8.4. What epsql can do?

UTF8 boxes

  • one new border style, five new border lines styles

Stored function's listing

  • should help with debuging

    postgres=# select test(10);
    ERROR: division by zero
    CONTEXT: PL/pgSQL function "test" line 3 at assignment

    postgres=# \lf test
    *** CREATE OR REPLACE FUNCTION public.test(a integer)
    *** RETURNS integer
    *** LANGUAGE plpgsql
    *** AS $function$
    1 declare b int;
    2 begin
    3 b := a/0;
    4 return a;
    5 end;
    *** $function$

Cursor's support

  • allows using data from database for statement parameters
    • \fetch
      postgres=# BEGIN;
      BEGIN
      postgres=# DECLARE c CURSOR FOR SELECT * FROM pg_database;
      DECLARE CURSOR
      postgres=# \fetch c \echo :datname :datcollate
      template1 cs_CZ.UTF-8
      postgres=# \fetch c \echo :datname :datcollate
      template0 cs_CZ.UTF-8
      postgres=# \fetch c \echo :datname :datcollate
      postgres cs_CZ.UTF-8
      postgres=# \fetch c \echo :datname :datcollate
      ERROR: (not available)
      postgres=# COMMIT;
      COMMIT
    • \fetchall
      postgres=# CREATE TABLE test1(a int); CREATE TABLE test2(a int);
      CREATE TABLE
      CREATE TABLE
      postgres=# BEGIN;
      BEGIN
      postgres=# DECLARE c CURSOR FOR SELECT tablename FROM pg_tables WHERE tablename LIKE 'test%';
      DECLARE CURSOR
      postgres=# \fetchall c DROP TABLE :tablename;
      DROP TABLE
      DROP TABLE
      postgres=# COMMIT;
      COMMIT
      postgres=#
I invite any ideas, comments.

Pavel

Labels:

Friday, February 20, 2009

8.3 migration helper

8.3 dropped some implicit casts. I didn't watch discus about it, but I am thinking so there are two reasons. First - it shows some programmer's mistakes, second - it warns against wrong written predicates. I can speak, so all it does well. When we migrated older application, we found lot of nonsense: typically numeric data are stored in varchar column or deprecated style (every constant is string literal) like:

any_numeric_column = 'any_number' ... -- deprecated
any_numeric_column = any_number ... -- good style

Peter Eisentraut wrote module for 8.3, that enable all disabled implicit casts. This helps with migration, but doesn't help with question, where disabled casts are used in application. I wrote function, that adds warning to theses casts:

CREATE OR REPLACE FUNCTION generator_81_casts()
RETURNS void AS $$
DECLARE
src varchar[] := '{integer,smallint,oid,date,double precision,real,time with time zone, time without time zone, timestamp with time zone, interval,bigint,numeric,timestamp without time zone}';
fn varchar[] := '{int4out,int2out,oidout,date_out,float8out,float4out,timetz_out,time_out,timestamptz_out,interval_out,int8out,numeric_out,timestamp_out}';
fn_name varchar;
fn_msg varchar; fn_body varchar;
BEGIN
FOR i IN array_lower(src,1)..array_upper(src,1) LOOP
fn_name := 'aux_cast_func_' || replace(src[i],' ','_') ||'_to_text';
fn_msg := '''using obsolete implicit casting from ' || src[i] || ' to text''';
fn_body := 'CREATE OR REPLACE FUNCTION '|| fn_name || '(' || src[i] ||') RETURNS text AS $_$ BEGIN RAISE WARNING '
|| fn_msg || ';RETURN textin(' || fn[i] || '($1)); END; $_$ LANGUAGE plpgsql IMMUTABLE';
EXECUTE fn_body;
-- for 8.1
--EXECUTE 'UPDATE pg_cast SET castfunc = ''' || fn_name || '''::regproc WHERE castsource = ''' || src[i] || '''::regtype AND casttarget = ''text''::regtype';
DROP CAST 'CREATE CAST (' || src[i] || ' AS text)';
EXECUTE 'CREATE CAST (' || src[i] || ' AS text) WITH FUNCTION ' || fn_name || '(' || src[i] || ') AS IMPLICIT';
END LOOP;
RETURN;
END;
$$ LANGUAGE plpgsql;
SELECT generator_81_casts();

Saturday, December 27, 2008

DB2 SQL Procedures on Postgres? Yes, why not?

I ported PL/pgPSM for PostgreSQL 8.4 this week. PL/pgPSM is implementation of SQL/PSM language to PostgreSQL space. SQL/PSM is procedural language from ANSI SQL standard. DB2 or MySQL use it. PL/pgPSM has some features from IBM implementation, but it isn't fully compatible. It isn't possible - PostgreSQL is little bit different than DB2. Relation between PL/pgPSM and SQL PL is like relation between PL/pgSQL and PL/SQL.

Installation

  1. download source code,
  2. extract it in PL directory in PostgreSQL 8.4 source code tree
  3. compile and install it
    [pavel@localhost Desktop]$ mv plpgpsm.tgz ../src/pgsql/src/pl/
    [pavel@localhost Desktop]$ cd ../src/pgsql/src/pl/
    [pavel@localhost pl]$ tar xvfz plpgpsm.tgz
    plpgpsm/Makefile
    plpgpsm/src/
    plpgpsm/src/pl_gram.c
    plpgpsm/src/pl_exec.c
    plpgpsm/src/gram.y
    plpgpsm/src/pl_funcs.c
    plpgpsm/src/plpgpsm.h
    plpgpsm/src/y.tab.h
    plpgpsm/src/sql/
    plpgpsm/src/sql/plpgpsm.sql
    plpgpsm/src/INSTALL.plpgpsm
    plpgpsm/src/Makefile
    plpgpsm/src/pl_handler.c
    plpgpsm/src/plerrcodes.h
    plpgpsm/src/scan.l
    plpgpsm/src/y.tab.c
    plpgpsm/src/pl_comp.c
    plpgpsm/src/expected/
    plpgpsm/src/expected/plpgpsm.out
    plpgpsm/src/pl_gram.h
    plpgpsm/src/pl_scan.c
    [pavel@localhost pl]$ cd plpgpsm/src/
    [pavel@localhost src]$ make all
    ...
    [pavel@localhost src]$ su
    Heslo:
    [root@localhost src]# make install
    /bin/sh ../../../../config/mkinstalldirs '/usr/local/pgsql/lib'
    /bin/sh ../../../../config/install-sh -c -m 755 plpgpsm.so '/usr/local/pgsql/lib/plpgpsm.so'

  4. register pl handler:
    [pavel@localhost src]$ psql template1
    psql (8.4devel)
    Type "help" for help.

    template1=# insert into pg_pltemplate
    select 'plpgpsm','t'::boolean, 't'::boolean, 'plpgpsm_call_handler','plpgpsm_validator','$libdir/plpgpsm',NULL
    from (
    select 'plpgpsm'
    except all
    select tmplname
    from pg_pltemplate
    ) a;
    INSERT 0 0

  5. Check it:

    [pavel@localhost src]$ pwd
    /home/pavel/src/pgsql/src/pl/plpgpsm/src
    [pavel@localhost src]$ make installcheck
    make -C ../../../../src/test/regress pg_regress
    make[1]: Entering directory `/home/pavel/src/pgsql/src/test/regress'
    make[1]: `pg_regress' is up to date.
    make[1]: Leaving directory `/home/pavel/src/pgsql/src/test/regress'
    ../../../../src/test/regress/pg_regress --inputdir=. --psqldir= --dbname=pl_regression --load-language=plpgpsm plpgpsm
    (using postmaster on Unix socket, default port)
    ============== dropping database "pl_regression" ==============
    DROP DATABASE
    ============== creating database "pl_regression" ==============
    CREATE DATABASE
    ALTER DATABASE
    ============== installing plpgpsm ==============
    CREATE LANGUAGE
    ============== running regression test queries ==============
    test plpgpsm ... ok

    =====================
    All 1 tests passed.
    =====================

Usage


  1. Read some documentation http://www.sqlpl-guide.com/ and http://www.pgsql.cz/index.php/SQL/PSM_Manual, http://www.pgsql.cz/index.php/MySQL5:_Gentle_introduction_to_stored_procedures_of_MySQL5
  2. Play with it and use it :)
    postgres=# CREATE OR REPLACE FUNCTION foo2(a integer)
    postgres-# RETURNS void AS
    postgres-# $$
    postgres$# BEGIN
    postgres$# DECLARE i integer DEFAULT 1;
    postgres$# WHILE i <= a
    postgres$# DO
    postgres$# PRINT i;
    postgres$# SET i = i + 1;
    postgres$# END WHILE;
    postgres$# END
    postgres$# $$ LANGUAGE plpgpsm;
    CREATE FUNCTION
    postgres=# select foo2(3);
    NOTICE: 1
    NOTICE: 2
    NOTICE: 3
    foo2 ------

    (1 row)

Tuesday, November 18, 2008

plpgsql and temp. tables

I tested speed of three possible styles of work with temp tables inside stored procedures. Usually I preferred checking of existence table to exception's trapping. I didn't expect it, but exception's trapping is the best of all.

create or replace function test1()
returns void as $$
begin
drop table if exists omega;
create temp table omega(a integer);
insert into omega values(10);
if exists(select * from omega) then end if;
end;
$$ language plpgsql;

create or replace function test2()
returns void as $$
begin
if exists(select * from pg_class where relname='omega' and pg_table_is_visible(oid)) then
delete from omega;
else
create temp table omega(a integer);
end if;
insert into omega values(10);
if exists(select * from omega) then end if;
end;
$$ language plpgsql;

create or replace function test3()
returns void as $$
begin
begin
delete from omega;
exception
when others then
create temp table omega(a integer);
end;
insert into omega values(10);
if exists(select * from omega) then end if;
end;
$$ language plpgsql;

test via pgbench

Test1:
transaction type: Custom query
scaling factor: 1
query mode: simple
number of clients: 10
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
tps = 339.780441 (including connections establishing)
tps = 340.172513 (excluding connections establishing)

Test2:
scaling factor: 1
query mode: simple
number of clients: 10
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
tps = 1891.021562 (including connections establishing)
tps = 1907.533096 (excluding connections establishing)

Test3:
transaction type: Custom query
scaling factor: 1
query mode: simple
number of clients: 10
number of transactions per client: 1000
number of transactions actually processed: 10000/10000
tps = 2664.756569 (including connections establishing)
tps = 2698.289177 (excluding connections establishing

Friday, August 29, 2008

updatable cursor's test

Hello

I had to update every row of table with an result of external function. It was possibility for testing of some patterns:
postgres=# create table testcursor(i integer primary key, v integer);
NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "testcursor_pkey" for table "testcursor"
CREATE TABLE
postgres=# insert into testcursor select i, 0 from generate_series(1,100000) g(i);
INSERT 0 100000
postgres=# analyze testcursor;
ANALYZE

/* external function sample */
create or replace function ext_fce(a integer, b integer)
returns int as $
declare r int;
begin
/* protect section */
begin
r := b;
exception when others then
r := null;
end;
return r;
end;
$$ language plpgsql;
1. test - standard update statement
postgres=# update testcursor set v = ext_fce(i, 30);
UPDATE 100000
Time: 4369,246 ms
2. test - update with updateable cursors
create or replace function testc2(_v integer)
returns void as $$
declare
c cursor for select i from testcursor;
_i integer;
begin
open c;
fetch c into _i;
while found loop
update testcursor set v = ext_fce(_i, _v)
where current of c;
fetch c into _i;
end loop;
close c;
end;
$$ language plpgsql;
postgres=# select testc2(20);
testc2
--------

(1 row)

Time: 8434,985 ms
3. test - update with PK
create or replace function testc3(_v integer)
returns void as $$
declare _i integer;
begin
for _i in select i from testcursor loop
update testcursor set v = ext_fce(_i,_v) where i = _i;
end loop;
end;
$$ language plpgsql;
postgres=# select testc3(30);
testc3
--------

(1 row)

Time: 9959,209 ms
Result: Using updateable cursor is about 90% slower than one statement. Usig pk is about 17% slower than updateable cursor. - So updateable cursors has sense and it is well to use it. But every iteration is much slower than one statement's update.

Wednesday, August 27, 2008

default parameters for PL functions

Hello

I completed one task - defaults for PL functions. Using is simple - it is same as Firebird's 2.x defaults.
postgres=# create or replace function x1(int = 1,int = 2,int= 3)
returns int as $$
select $1+$2+$3;
$$ language sql;
CREATE FUNCTION
postgres=# select x1();
x1
----
6
(1 row)

postgres=# select x1(10);;
x1
----
15
(1 row)

postgres=# select x1(10,20);
x1
----
33
(1 row)

postgres=# select x1(10,20,30);
x1
----
60
(1 row)
This is first step before named parameters feature - and less controversy. Second step will be difficult - there are two opinions about named parameter's syntax: variant a) using Oracle's syntax name => expression and variant b) use own syntax based on keyword "AS" expression AS name. I prefer variant @a - I thing so it's more readable (SQL use AS for labeling). Variant @b is safe from compatibility views. There was discussion on pg_hackers - without any conclusion. So I hope so at least defaults will be committed.

bye
Pavel

Labels: ,

Monday, August 25, 2008

Using cursors for generating cross tables

I am working on procedure support in PostgreSQL more than one year. Missing procedures is one disadvantage of PostgreSQL. Because procedures are not called from SELECT statement, then it couldn't respect some rules like functions. Procedures are mainly used for explicit transaction controlling and for generating mutable results (like dynamic record set or stacked record set). Dynamic record set is interesting feature when we nothing know about result set's columns. It's typical for cross tables. Because PostgreSQL doesn't support procedures, we cannot return dynamic (mutable number of columns) queries directly, but we can return dynamic cursor. This method of generating cross tables is inspired by Roland Bauman's http://rpbouman.blogspot.com blog.
Cross table is based on query's pattern:
SELECT shop,
SUM(CASE gender WHEN 'f' THEN salary ELSE 0 END) AS f,
SUM(CASE gender WHEN 'm' THEN salary ELSE 0 END) AS m,
SUM(salary) AS total
FROM employees INNER JOIN shops USING (shop_id)
GROUP BY shop

for data:
CREATE TABLE employees (
id serial PRIMARY KEY,
shop_id int,
gender char(1),
name varchar(32),
salary int
);

CREATE TABLE shops (
id serial PRIMARY KEY,
shop varchar(32)
);

INSERT INTO shops(shop) VALUES('Zurich'),('New York'),('London');

INSERT INTO employees (shop_id, gender, name, salary)
VALUES
(1, 'm', 'Jon Simpson', 4500),
(1, 'f', 'Barbara Breitenmoser',4700),
(2, 'f', 'KirstenRuegg',5600),
(3, 'm', 'Ralp Teller',5100),
(3, 'm', 'Peter Jonson',4700);
I write function, that generate necessary SELECT statement and open dynamic cursor.
CREATE OR REPLACE FUNCTION do_cross_cursor(dimx_name varchar,
dimx_source varchar, dimy_name varchar,
dimy_source varchar, expr varchar)
RETURNS refcursor AS $$
DECLARE
col_list text[] := '{}';
query text;
r RECORD;
result refcursor := 'result';
BEGIN
FOR r IN EXECUTE 'SELECT DISTINCT '
|| dimx_name || '::text AS val ' || dimx_source
LOOP
col_list := array_append(col_list, 'SUM(CASE ' || dimx_name
|| ' WHEN ' || quote_literal(r.val) || ' THEN ' || expr
|| ' ELSE 0 END) AS ' || quote_ident(r.val) || '');
END LOOP;
query := 'SELECT ' || dimy_name || ', '
|| array_to_string(col_list, ',')
|| ', SUM(' || expr || ') AS Total '
|| dimy_source || ' GROUP BY ' || dimy_name;
OPEN result NO SCROLL FOR EXECUTE query;
RETURN result;
END;
$$ LANGUAGE plpgsql STRICT;
Because cursors should be used only in transaction, I have to use explicit transaction:
BEGIN;
SELECT do_cross_cursor('gender', 'FROM employees','shop',
'FROM employees e JOIN shops s ON s.id = e.shop_id',
'salary');
FETCH ALL FROM result;
END;

BEGIN;
SELECT do_cross_cursor('shop', 'FROM shops','gender',
'FROM employees e JOIN shops s ON s.id = e.shop_id',
'salary');
FETCH ALL FROM result;
END;
There is result:
postgres=# BEGIN;
BEGIN
postgres=# SELECT do_cross_cursor('gender', 'FROM employees',
'shop',
'FROM employees e JOIN shops s ON s.id = e.shop_id', 'salary');
do_cross_cursor
-----------------
result
(1 row)

postgres=# FETCH ALL FROM result;
shop | m | f | total
----------+------+------+-------
New York | 0 | 5600 | 5600
Zurich | 4500 | 4700 | 9200
London | 9800 | 0 | 9800
(3 rows)

postgres=# END;BEGIN;
COMMIT
BEGIN
postgres=# SELECT do_cross_cursor('shop', 'FROM shops','gender',
'FROM employees e JOIN shops s ON s.id = e.shop_id', 'salary');
do_cross_cursor
-----------------
result
(1 row)

postgres=# FETCH ALL FROM result;
gender | New York | Zurich | London | total
--------+----------+--------+--------+-------
m | 0 | 4500 | 9800 | 14300
f | 5600 | 4700 | 0 | 10300
(2 rows)

postgres=# END;
COMMIT