I've been working on putting together a nice base for my theological library for the past few years (I still have a long way to go). I've put a lot of time and thought into its composition (and my wife says a lot of money too). Along the way I've learned a few things so I wanted to put a post together to provide some suggestions on how to build one from scratch or improve an existing library.
1. Preview before you buy
Books, especially commentaries, are expensive. Try to preview them in some way before you buy them. This will save you money and disappointment. There are a few ways you can do this. If you live near a good local theological library, spend a few hours there reading. Even your local public library may have what you're looking for. Also, follow publishers' blogs. They often have previews of a chapter or two of forthcoming books. Another way to get a look inside a book is Google Books, which has previews or the full text of some books available.
2. Think about it
There are so many good books to buy. So many! Often times I've learned about a book and gotten really excited about a book and wanted to buy it only to have my initial excitement wear off a few days later. To combat this, I've instituted a waiting period before I buy anything to guard myself against impulse purchases. When I find the book I like I put it on my amazon wish list, then several days later I go back and compare it to other books on my wish list. If I still want it more than any other book on my wish list (which I keep up to date and use their priority rankings) in the same price range, then I'll pull the trigger.
3. You need more than commentaries
A good theological library should contain commentaries, bible dictionaries, introductory books covering portions of Scripture (NT, OT, Paul, the Gospels, the prophets, etc.) both of the historical critical type and the theological type, systematic theology texts, books of historical theology, lexicons and grammars, books on church history, and some basic references on the Apocrypha, Pseudepigrapha, and the Apostolic Fathers. You also should buy important monographs within each of these fields that provide in-depth studies on various matters.
4. Don't buy it just because...
it's cheap or it's by an author you like. I've seen the former mistake made often, especially related to commentaries. You have a lacuna in your library so when you see a book or series on sale that addresses your need you jump on it. Hold your horses, especially if you don't know much about the book/series you're buying. Series in particular tend to be very uneven (thus I don't recommend buying a whole commentary series unless you already have a very deep library and have lots of money) so your great deal may not end up solving your problem. As I mentioned too, not all works by the same author are of equal quality, especially if they're churning books out at record pace.
5. Be diverse
No tradition gets everything right and there's great benefit to seeing Scripture from different angles. Especially when buying commentaries, try to buy books from several different positions. Buy Reformed and Arminiain; Evangelical, Mainline Protestant, and Catholic. Be very, very intentional about this. For example, if you're reformed and you want to buy four commentaries on Romans, avoid the temptation of buying Morris, Moo, Schreiner, and Cranfield. They're great commentaries, but you'll only be looking at Romans from one general perspective. Consider going with Moo, Jewett, Wright, and Fitzmyer; or Cranfield, Dunn, Schreiner, and Keener.
In addition to buying from a variety of theological perspectives, vary the style of commentary that you purchase. You need at least one commentary written within the last twenty years that comments on the original language. I also think that buying a commentary that takes a more theological approach is a necessity as well. Two commentary series, the Two Horizons and the Brazos Theological Commentary series are provide rich theological exegesis. They're new kids on the block, so many books of the Bible aren't covered by either of these series.
Also consider the approach taken to study the text. For example, on the gospels you don't a bunch of commentaries that focus heavily on form and redaction criticism, even if you're a big fan of it. Look at some that do narrative criticism, or that are written by scholars who focus on the oral traditions behind the text.
6. Don't neglect older works
There are a lot of good works out there by dead guys. Read them! IVP and Crossway in particular have been helpful in bringing the resources of the past to today's readers. Crossway has a series of commentaries (Crossway Classic Commentaries) written by the greats of the Reformation tradition. IVP, as of late, has been publishing a lot of texts from the early church, under the label 'Ancient Christian.' These include their Ancient Christian Commentary Series, which cobbles together quotes from various early church commentators, to full single author commentaries and books on various doctrinal topics. Hopefully someone will fill in the gap and produce English translations of Medieval works. There also is a lot of good stuff out there in the area of theology, whether it be Aquinas, Calvin, Barth, Wesley, Augustine, or others. Get your hands on their stuff and read it!
7. Buy more than you think you need because you'll never know when you need it
I know that good reference books are expensive but you need to have enough books on your shelf that when you have a question about a particular passage or a significant topic you have a place or two to start looking for answers. You never know when that question will pop up, so it's best to be prepared in advance. A short story to illustrate. This past Fall, I taught a class in my church on Daniel. I wanted to spend a little time studying the Additions to Daniel in the Apocrypha. It just so happened that six months prior I had purchased Invitation to the Apocrypha. It gave me a helpful resource for my study.
8. It's not just about you
When building your library, try to be other focused. Encourage people to borrow from you. Don't be afraid to give away a book that has been helpful to you. Possessions can lead to covetousness and hoarding, and this certainly includes books. Don't let your library oppress you, but use it to its fullest for God's glory by helping those who aren't so fortunate to have such an awesome library!