Performance on despatches

PostgreSQL performing huge updates

Sun, 06 Nov 2011 12:45:41 +0000

PostgreSQL is a pretty powerful database server and will work with almost any settings thrown at it. It is really good at making do with what it has and performing as it is asked.

We recently found this as we were trying to update every row in a table that had over eight million entries. We found in the first few tries that the update was taking over 24 hours to complete which was far too long for an update script.

Our investigation of this led us to the pgsql_tmp folder and the work_mem configuration parameter.

When the query was being executed, we checked the pgsql_tmp folder to see how was space being utilised in there. We already knew about the pgsql table from past experience. We had a server running out of disk space and rapidly. We had narrowed it down into this folder. In cancelling the query referenced by the tmp files in here, we were able to free up literally gigabytes of disk space...

We had found roughly half a gig of temporary files in here. This led us to investigate the configuration file.

The one parameter that stuck out was work_mem which was set to a default of 1mb which I guess might make sense under most circumstances but not in this one. According to the postgresql documentation

work_mem (integer)

Specifies the amount of memory to be used by internal sort operations and hash tables before switching to temporary disk files. The value is defaults to one megabyte (1MB). Note that for a complex query, several sort or hash operations might be running in parallel; each one will be allowed to use as much memory as this value specifies before it starts to put data into temporary files. Also, several running sessions could be doing such operations concurrently. So the total memory used could be many times the value of work_mem; it is necessary to keep this fact in mind when choosing the value. Sort operations are used for ORDER BY, DISTINCT, and merge joins. Hash tables are used in hash joins, hash-based aggregation, and hash-based processing of IN subqueries.

This would tell us that the total memory usage with work_mem could be several times the value set here and setting it to half a gig would probably be a terrible idea for a heavily utilised production server. However, for the migration process when we need to update over 8,000,000 rows, it might be a good temporary fix.

After updating the work_mem to 512mb, we found that no more tmp files were created and the whole thing was done in memory.

When updating so many rows, there area a few other things to consider.

Firstly, autovacuum will likely kick in several times to vacuum the table. You'll probably want to disable this for the duration of the update statement and run a vacuum afterwards.

```sql --disable auto vacuum ALTER TABLE sometable SET ( autovacuum_enabled = false, toast.autovacuum_enabled = false ); ```

You can switch autovacuum back on after the update statement has completed

```sql --enable auto vacuum ALTER TABLE sometable SET ( autovacuum_enabled = true, toast.autovacuum_enabled = true ); ```

A few other things you want to take a look at are the

fsync parameter (I usually have this set to off anyway since the servers are pratically fully redundant)
checkpoint_segments: I changed this to roughly 5 times the original value (check the log to see if it says that its checkpointing too often)
checkpoint_completion_target: I changed this to 0.9

With all of these updates, we were able to bring the total time of the update down to a few hours.

Making Twitter Better

Mon, 09 Mar 2009 15:01:40 +0000

I think that twitter is a fantastic service and has a bright future. However, like a lot of new things, the question of whether it will flourish or perish is really all down how the growth is managed, planned and executed.

I should point out that I don’t know the people at twitter at all and is very much an outsiders opinion. I have been running a business for about nine years, and while it is of nowhere near the success of twitter, I’ve definitely learned some hard lessons. I am not complaining - I am however, voicing some ideas on how things could be made better.

My experience also includes working very closely with megabus.com, which grew from a fledgling website 6 years ago to what it is today servicing over a 100,000 visitors every day.

My gut instinct about Twitter is that the guys and gals are working hard to delivery one really good service really well. However, it is of a size now where service delivery should be happening in the background with little or no effort.

When megabus.com first launched and over the first couple of years, we spent a lot of time managing the hardware, software and processes till we got it right. It went through a dramatic re-architecture in 2005 and since then, the management time has dropped dramatically.

To take twitter to the next level so that it can be bigger than facebook, in my opinion, requires twitter to a lot of things:

Reliability & Performance

I don’t know the architecture / infrastructure of twitter but having used it fairly heavily over the last few days, have noticed intermittent outages. This has to be solved. Not just in the short term, but in the medium and long term. Twitter has to be a service that just works. All websites suffer glitches and outages but the mean time to failure needs to be a lot higher and it should be cheap and cost effective to scale.

TwitApplications

There are a lot of services and applications that link into twitter. I consistently use tweetburner, tweetdeck and have looked at / considered a range of other services / applications. While the wiki page can point someone in the right direction. This needs to be integrated better into twitter itself

Facebook really took off and removed bebo and myspace as competitors, in my opinion the day it introduced facebook applications.

It should be a different process from facebook as facebook applications are of a different breed and different target market. Twitter simply needs to make it easier for applications to integrate in to solve two problems

Easy launchpad to add them in and use them
Remove the need to provide the twitter username/password in other websites. I currently have to do this with tweetburner to post directly which makes me very uncomfortable.

Accessibility

I am not talking about makes it easier for people with disabilities to access the site. I am talking about people who are not technically savvy or more importantly twitter savvy.

I joined twitter a while back and just felt a bit lost. There was no guidance as to what a tweet was, what it meant to be a follower or what it meant for people to follow you.

It took an article on a magazine explaining it to make it easier for me to understand and re-boot my twitter life.

Help & Support are good and useful but it should not be necessary if the help and support is present throughout the site. Facebook does this well and makes it easy to learn and do new things. It does not need to be idiot proof but it does need to have just enough information for a newbie to get started.

There are numerous blogs, articles and websites that cover this information but that means that someone has to spend enough effort getting out there and finding out.

This can be difficult when you don’t know what you are searching for as well.

Functional Integrations

There are several integrations that would be useful. There are websites that do some of these things but it would be useful to have them integrated within the site. Examples include:

Easy way to see the last tweet of all the people you are following / your followers
Popularity of the people you are following / your followers
Group people, so that you can follow people who blog about different things but read them together

Conclusion

From my perspective, this is of course a starting point, the tip of the iceberg. Twitter is involved in a lot of new things but without the soft aspect, I think it is making its life harder than it has to be to get the masses.