Before you actually engage with them, here are a few important questions you should ask:
1. Does the SEO agency understand my business?
Or again more specifically, do the people actually working on my account understand what my organisation does, its products or services and what it's unique proposition or selling point is?
There will almost certainly be some initial familiarisation with your offering or processes, but first check that this agency gets what you do and has a firm understanding of who your competitors are.
Note: The counter to this issue however is when you hire an SEO agency that knows your industry very well and already has a number of your competitors as clients in this market sector... are they really able to provide you with a unique and perhaps innovative approach to on-page and off-site optimisation?
Or more specifically, how much of what you pay for is agency 'padding' in the form of 'Project Management', 'Account Management' or even worse... 'Administration & Reporting'?
(Note, most SEO tools these days have quite decent automated reporting functions. So sending out a regular report is just a case of configuring the reporting service once).
In one situation I saw last year, where my agency won the SEO business, The outgoing agency managed to fill over 60% of its monthly SEO retainer with non-specialist staff. Nice work if you can get it....
3. Who is actually doing the work?
It's a pretty good bet that you had a smart(ish) new business person put together the proposal that you accepted. Or if the agency is a smaller one, then it may well be the owner or other senior person that wrote the document that won them the work. But will this person be the one actually working on your account day-to-day or will it be a junior person they may not even have mentioned in their credentials? My guess is that in most SEO agencies it will be the latter that does the hard graft most of the time (not the 'Head of Search' or 'Head of SEM Services' you were promised)
Note: If it is someone you never get to speak to, then reconsider hiring them. And if it's 'someone in their [not this country] office' then get concerned quickly, really quick....
4. What tools & techniques do you use?
Some SEO agencies like to keep the tools and techniques they use a secret to their clients. I guess they feel it adds an air of mystery to the complex art of search engine optimisation. As far as tools go, there are a few good ones out there that the majority of agencies use for most of their clients. Also make sure that you are not being charged extra for these tools, the costs for them should be included in your retainer.
Note: Some of these tools use propriety indexing technology to work, whereas others need to link to your own site's Webmaster Tools accounts. Neither is wrong, but be prepared to grant them access in the same sort of ways you've granted them access to your website analytics package.
As for the techniques used... you should have full transparency about what they are doing and the rationale for doing it. However, if they mention the act of buying links... run a mile!
5. What do you want me to do next?
Getting started with a new agency is usually a process of learning, testing, evaluating and refining. Expect the agency to ask to speak with other stakeholders or 3rd parties in your business (e.g. PR company, website development agency and product / catalogue managers if you have an eCommerce site). Having an SEO firm that is not just technically competent, but has decent organisational skills can be a rare find. Also make sure that you have regular review sessions booked in the diaries. Even if these are done over the phone / Skype or webex... your business, the competition and most definitely the search engines change all the time.