Database-Systems on despatcheshttps://icle.es/tags/database-systems/Recent content in Database-Systems on despatchesHugoenWed, 18 Mar 2026 15:13:17 +0000PostgreSQL performing huge updateshttps://icle.es/2011/11/06/postgresql-performing-huge-updates-1106/Sun, 06 Nov 2011 12:45:41 +0000https://icle.es/2011/11/06/postgresql-performing-huge-updates-1106/<p>PostgreSQL is a pretty powerful database server and will work with almost any settings thrown at it. It is really good at making do with what it has and performing as it is asked.</p> <p>We recently found this as we were trying to update every row in a table that had over eight million entries. We found in the first few tries that the update was taking over 24 hours to complete which was far too long for an update script.</p> <p>Our investigation of this led us to the pgsql_tmp folder and the work_mem configuration parameter.</p> <p>When the query was being executed, we checked the pgsql_tmp folder to see how was space being utilised in there. We already knew about the pgsql table from past experience. We had a server running out of disk space and rapidly. We had narrowed it down into this folder. In cancelling the query referenced by the tmp files in here, we were able to free up literally gigabytes of disk space...</p>PostgreSQL is a pretty powerful database server and will work with almost any settings thrown at it. It is really good at making do with what it has and performing as it is asked.

We recently found this as we were trying to update every row in a table that had over eight million entries. We found in the first few tries that the update was taking over 24 hours to complete which was far too long for an update script.

Our investigation of this led us to the pgsql_tmp folder and the work_mem configuration parameter.

When the query was being executed, we checked the pgsql_tmp folder to see how was space being utilised in there. We already knew about the pgsql table from past experience. We had a server running out of disk space and rapidly. We had narrowed it down into this folder. In cancelling the query referenced by the tmp files in here, we were able to free up literally gigabytes of disk space...

We had found roughly half a gig of temporary files in here. This led us to investigate the configuration file.

The one parameter that stuck out was work_mem which was set to a default of 1mb which I guess might make sense under most circumstances but not in this one. According to the postgresql documentation

work_mem (integer)

Specifies the amount of memory to be used by internal sort operations and hash tables before switching to temporary disk files. The value is defaults to one megabyte (1MB). Note that for a complex query, several sort or hash operations might be running in parallel; each one will be allowed to use as much memory as this value specifies before it starts to put data into temporary files. Also, several running sessions could be doing such operations concurrently. So the total memory used could be many times the value of work_mem; it is necessary to keep this fact in mind when choosing the value. Sort operations are used for ORDER BY, DISTINCT, and merge joins. Hash tables are used in hash joins, hash-based aggregation, and hash-based processing of IN subqueries.

This would tell us that the total memory usage with work_mem could be several times the value set here and setting it to half a gig would probably be a terrible idea for a heavily utilised production server. However, for the migration process when we need to update over 8,000,000 rows, it might be a good temporary fix.

After updating the work_mem to 512mb, we found that no more tmp files were created and the whole thing was done in memory.

When updating so many rows, there area a few other things to consider.

Firstly, autovacuum will likely kick in several times to vacuum the table. You'll probably want to disable this for the duration of the update statement and run a vacuum afterwards.

```sql --disable auto vacuum ALTER TABLE sometable SET ( autovacuum_enabled = false, toast.autovacuum_enabled = false ); ```

You can switch autovacuum back on after the update statement has completed

```sql --enable auto vacuum ALTER TABLE sometable SET ( autovacuum_enabled = true, toast.autovacuum_enabled = true ); ```

A few other things you want to take a look at are the

  • fsync parameter (I usually have this set to off anyway since the servers are pratically fully redundant)
  • checkpoint_segments: I changed this to roughly 5 times the original value (check the log to see if it says that its checkpointing too often)
  • checkpoint_completion_target: I changed this to 0.9

With all of these updates, we were able to bring the total time of the update down to a few hours.

]]>Tracking progress of an update statementhttps://icle.es/2011/11/02/tracking-progress-of-an-update-statement-1101/Wed, 02 Nov 2011 19:59:02 +0000https://icle.es/2011/11/02/tracking-progress-of-an-update-statement-1101/<p>Sometimes there is a need to execute a long running update statement. This update statement might be modifying millions of rows as was the case when we went hunting for a way to track the progress of the update. Hunting around took us to <a href="http://archives.postgresql.org/pgsql-admin/2002-07/msg00286.php">http://archives.postgresql.org/pgsql-admin/2002-07/msg00286.php</a> In our particular case, we are using postgresql but this should work with any database server that provides sequences. Our original sql was of the form:</p> ```sql update only table1 t1 set amount = t2.price from table2 t2 where t1.id = t2.id; ``` <p>There is of course now way of figuring out how many rows had been updated already. The first step was to create a sequence</p> ```sql CREATE TEMPORARY SEQUENCE seq_progress START 1; ```Sometimes there is a need to execute a long running update statement. This update statement might be modifying millions of rows as was the case when we went hunting for a way to track the progress of the update. Hunting around took us to http://archives.postgresql.org/pgsql-admin/2002-07/msg00286.php In our particular case, we are using postgresql but this should work with any database server that provides sequences. Our original sql was of the form:

```sql update only table1 t1 set amount = t2.price from table2 t2 where t1.id = t2.id; ```

There is of course now way of figuring out how many rows had been updated already. The first step was to create a sequence

```sql CREATE TEMPORARY SEQUENCE seq_progress START 1; ```

We can then use this sequence in the update statement to ensure that each row updated also increments the sequence

```sql update only table1 t1 set amount = t2.price from table2 t2 where nextval('seq_progress') != 0 and t1.id = t2.id; ```

Once the query is running, you can open another connection to the database. To get an indication of how far it has got, you can just run the following

```sql select nextval('seq_progress'); ```

Bear in mind that this will also increment it by 1 but if you have millions of rows which is really the only case in which this would be useful, a few additional increments is hardly going to make a difference.

Good luck and have fun!

]]>Database Systems Comparedhttps://icle.es/2009/03/10/database-systems-compared/Tue, 10 Mar 2009 16:00:21 +0000https://icle.es/2009/03/10/database-systems-compared/<p>My first experiences of a computer started with <a href="http://en.wikipedia.org/wiki/DBase" title="Dbase on Wikipedia">DBase III+</a>which is now <a href="http://www.dbase.com/" title="dBASE">dBASE</a>, then went on to <a href="http://en.wikipedia.org/wiki/FoxPro_2" title="Foxpro 2 on Wikipedia">Foxpro</a>, now <a href="http://msdn.microsoft.com/en-us/vfoxpro/bb190288.aspx" title="Microsoft Visual Foxpro">Microsoft Visual Foxpro</a>.</p> <p>I have since used:</p> <ul> <li><a href="http://www.filemaker.co.uk/" title="Filemaker Pro">Filemaker Pro</a>,</li> <li><a href="http://office.microsoft.com/en-us/access/default.aspx" title="Microsoft Access">Microsoft Access</a>,</li> <li><a href="http://www.microsoft.com/sqlserver/2008/en/us/default.aspx" title="Microsoft SQL Server">Microsoft SQL Server</a>,</li> <li><a href="http://www.mysql.com/" title="MySQL">MySQL</a>,</li> <li><a href="http://www.postgresql.org/" title="PostgreSQL">PostgreSQL</a>,</li> <li><a href="http://www.sqlite.org/" title="SQLite">SQLite</a> and</li> <li><a href="http://hsqldb.org/" title="HSQLDB">HSQLDB</a>.</li> </ul> <p>I have not yet used:</p> <ul> <li><a href="http://www.ibm.com/software/data/db2/" title="IBM DB2">IBM DB2</a>,</li> <li><a href="http://www.oracle.com/index.html" title="Oracle">Oracle</a>.</li> </ul> <p><a href="http://en.wikipedia.org/wiki/Comparison_of_relational_database_management_systems" title="Compare DB Systems">Wikipedia has a list of database systems</a>.</p>My first experiences of a computer started with DBase III+which is now dBASE, then went on to Foxpro, now Microsoft Visual Foxpro.

I have since used:

I have not yet used:

Wikipedia has a list of database systems.

Having worked with this range of database systems and having done copious amounts of research into DB2, Oracle and other DB systems I have not mentioned, I like answering the age old questions. Which is the best database system?

Ah! if only it was that simple. There is no database system that is appropriate for any given requirement. But then, if you have been in the technology sector long enough, you would already know that. It's all about using the right tool for the job.

I separate these systems into two broad categories and Oracle. There are the Desktop based database systems:

  • DBase
  • Foxpro
  • SQLite
  • HSQLDB
  • Filemaker Pro
  • Microsoft Access
  • MySQL

DBase, FoxPro, Filemaker Pro and Microsoft Access are essentially a GUI frontend that has a database backing.

Access is the best choice for this purpose under the majority of circumstances. Filemaker Pro is relevant in some. The usual reason to use DBase or FoxPro is simply that the developer is used to it. This is not a good enough reason.

I have used DBase III+ for developing an office management suite back in 1994. I have since used Filemaker Pro to develop a simple contact management database in 1998, Microsoft Access to develop a patient management system for a clinic.

SQLite, HSQLDB and MySQL are database engines that are to be utilised by popping a frontend on top; sometimes the frontend is Microsoft Access. Microsoft Access can also be used for its database engine.

Access is usually the worst choice for this except as a stopgap. There are exceptions to this. One is for a web frontend if the site is not too busy and its running on a microsoft platform. You don't have to go to the hassle of installing anything on the server. The drivers will take care of it all.

HSQLDB becomes an obvious choice for a light java based application and SQLite for any other lightweight applications.

MySQL is substantially more powerful and scales a lot better. I include it in this section because it is a server grade database system that can also work well in a desktop environment.

I have used Access for several web based systems and I have used HSQLDB for unit testing hibernate and for a quick and dirty MP3 library that linked into musicBrainz. I have used SQLite in passing to be utilised by open source products.

I have used MySQL with an Access frontend as a management suite for a website as well.

And we have the server based database systems:

  • MySQL
  • Microsoft SQL Server
  • IBM DB2
  • PostgreSQL

MySQL was used as the backed database system for the edFringe.com website. This was the perfect choice since the most important requirement was speed. Particuarly with the Query Cache and Master Slave replication, MySQL was the best choice.

SQL Server was used as the backend system for an online course for the Scottish Enterprise around 1999/2000. While MySQL would have been a good choice this, it was not of production quality at the time.

We have also used Ms SQL Server for an insurance company since all the infrastructure was based on Windows and PostgreSQL did not have a viable Windows version at the time.

We use PostgreSQL for megabus. While speed is absolutely critical, it is a ticketing system which means that transactionality is absolutely critical.

While MySQL now has transactionality with innodb, it is still nowhere near as good as the transactionality provided by PostgreSQL through MVCC (Multi-version Concurrency Control). We could have used Ms SQL Server but the cost savings are dramatic.

To summarise, each system has a specific use, specific strengths and weaknesses and which should be used is highly dependent on what it is to be used for. I am hopeful that the summary of what we have used each of these systems for us useful in determining which one is best placed to solve any specific problem :-D

We have not yet used Oracle and it was a strong contender for megabus but the serious heavyweight functionality provided by Oracle comes at a price and it is not yet a cost effective option.

]]>